[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:42:08 dist-test-slave-smpn krb5kdc[7804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:42:08 dist-test-slave-smpn krb5kdc[7804](info): set up 2 sockets
Sep 29 18:42:08 dist-test-slave-smpn krb5kdc[7804](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 18:42:09.584052 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:42:09 dist-test-slave-smpn krb5kdc[7804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:42:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:42:09.702642 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:40231
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:40231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:09.810496 7817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:09.810804 7817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:09.810879 7817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:09.814982 7817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:09.815088 7817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:09.815133 7817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:09.815171 7817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:09.815205 7817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:09.820674 7817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:40231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:40231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:09.821846 7817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:09.822934 7817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:42:09.831665 7817 server_base.cc:1034] running on GCE node
W20260929 18:42:09.831735 7823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:09.831745 7825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:09.831689 7822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:09.832456 7817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:09.833515 7817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:09.834662 7817 hybrid_clock.cc:648] HybridClock initialized: now 1790707329834641 us; error 34 us; skew 500 ppm
Sep 29 18:42:09 dist-test-slave-smpn krb5kdc[7804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707329, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:09.845121 7817 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:42:09.846087 7817 webserver.cc:460] Webserver started at http://127.7.157.126:43221/ using document root <none> and password file <none>
I20260929 18:42:09.846688 7817 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:09.846751 7817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:09.846998 7817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:09.849244 7817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "7d5d737fd77a42c08c3f6c1b7c72b2bf"
format_stamp: "Formatted at 2026-09-29 18:42:09 on dist-test-slave-smpn"
server_key: "3b420764460a9044a00a96bfb0fba78b"
server_key_iv: "ed1d65ae2d85f3f7bc920edd710cb94d"
server_key_version: "encryptionkey@0"
I20260929 18:42:09.849817 7817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "7d5d737fd77a42c08c3f6c1b7c72b2bf"
format_stamp: "Formatted at 2026-09-29 18:42:09 on dist-test-slave-smpn"
server_key: "3b420764460a9044a00a96bfb0fba78b"
server_key_iv: "ed1d65ae2d85f3f7bc920edd710cb94d"
server_key_version: "encryptionkey@0"
I20260929 18:42:09.853855 7817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:42:09.856503 7832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:09.857793 7817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:09.857906 7817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "7d5d737fd77a42c08c3f6c1b7c72b2bf"
format_stamp: "Formatted at 2026-09-29 18:42:09 on dist-test-slave-smpn"
server_key: "3b420764460a9044a00a96bfb0fba78b"
server_key_iv: "ed1d65ae2d85f3f7bc920edd710cb94d"
server_key_version: "encryptionkey@0"
I20260929 18:42:09.858016 7817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:09.878006 7817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:09.880923 7817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:09.881102 7817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:09.887768 7817 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:40231
I20260929 18:42:09.887785 7884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:40231 every 8 connection(s)
I20260929 18:42:09.888702 7817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:09.891383 7885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:09.896916 7885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf: Bootstrap starting.
I20260929 18:42:09.898582 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 7817
I20260929 18:42:09.898711 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:09.899531 7885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:09.900039 7797 external_mini_cluster.cc:1428] Setting key 11682d4e6c20ba6e8a20bc959ad18da1
I20260929 18:42:09.900321 7885 log.cc:824] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:09.902956 7885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf: No bootstrap required, opened a new log
I20260929 18:42:09.907644 7885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5d737fd77a42c08c3f6c1b7c72b2bf" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40231 } }
I20260929 18:42:09.907893 7885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:09.907967 7885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d5d737fd77a42c08c3f6c1b7c72b2bf, State: Initialized, Role: FOLLOWER
I20260929 18:42:09.908427 7885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5d737fd77a42c08c3f6c1b7c72b2bf" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40231 } }
I20260929 18:42:09.908555 7885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:09.908627 7885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:09.908722 7885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:09.909684 7885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5d737fd77a42c08c3f6c1b7c72b2bf" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40231 } }
I20260929 18:42:09.910015 7885 leader_election.cc:302] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d5d737fd77a42c08c3f6c1b7c72b2bf; no voters:
I20260929 18:42:09.910306 7885 leader_election.cc:288] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:09.910383 7890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:09.910569 7890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [term 1 LEADER]: Becoming Leader. State: Replica: 7d5d737fd77a42c08c3f6c1b7c72b2bf, State: Running, Role: LEADER
I20260929 18:42:09.910863 7890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5d737fd77a42c08c3f6c1b7c72b2bf" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40231 } }
I20260929 18:42:09.911648 7885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:09.913213 7892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d5d737fd77a42c08c3f6c1b7c72b2bf. Latest consensus state: current_term: 1 leader_uuid: "7d5d737fd77a42c08c3f6c1b7c72b2bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5d737fd77a42c08c3f6c1b7c72b2bf" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40231 } } }
I20260929 18:42:09.913335 7892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:09.913710 7891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d5d737fd77a42c08c3f6c1b7c72b2bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5d737fd77a42c08c3f6c1b7c72b2bf" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40231 } } }
I20260929 18:42:09.914151 7891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:09.914213 7905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:09.917265 7905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 18:42:09 dist-test-slave-smpn krb5kdc[7804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:09.922914 7905 catalog_manager.cc:1347] Generated new cluster ID: d140f5ce721d431bba9325e582bbfa50
I20260929 18:42:09.922976 7905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:09.925573 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:09.903629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54626 (local address 127.7.157.126:40231)
0929 18:42:09.904064 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:09.904080 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:09.905402 (+ 1322us) server_negotiation.cc:407] Connection header received
0929 18:42:09.908113 (+ 2711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:09.908127 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:09.908426 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:09.908742 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:09.911317 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:09.912638 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:09.914306 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:09.914794 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:09.914907 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:09.919678 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:09.919684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:09.919700 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:09.919731 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:09.921990 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:09.922608 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:09.922613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:09.922619 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:09.922765 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:09.923156 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:09.923159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:09.923160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:09.923660 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:09.923849 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:09.924486 (+ 637us) server_negotiation.cc:299] Negotiation successful
0929 18:42:09.924751 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":128,"threads_started":1}
I20260929 18:42:09.931293 7905 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:09.932440 7905 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:09.942641 7905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d5d737fd77a42c08c3f6c1b7c72b2bf: Generated new TSK 0
I20260929 18:42:09.943545 7905 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:42:09.972111 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:40231
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:10.085559 7911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:10.085825 7911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:10.085867 7911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:10.089963 7911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:10.090024 7911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:10.090082 7911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:42:10.095230 7911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:40231
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:10.096223 7911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:10.097222 7911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:10.106581 7916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:10.106642 7917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:10.106760 7911 server_base.cc:1034] running on GCE node
W20260929 18:42:10.106588 7919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:10.107273 7911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:10.107910 7911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:10.109068 7911 hybrid_clock.cc:648] HybridClock initialized: now 1790707330109035 us; error 43 us; skew 500 ppm
Sep 29 18:42:10 dist-test-slave-smpn krb5kdc[7804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707330, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:10.111856 7911 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:42:10.112818 7911 webserver.cc:460] Webserver started at http://127.7.157.65:42071/ using document root <none> and password file <none>
I20260929 18:42:10.113389 7911 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:10.113456 7911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:10.113659 7911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:10.115892 7911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "56b9c02a507d40de9340c99f8774c456"
format_stamp: "Formatted at 2026-09-29 18:42:10 on dist-test-slave-smpn"
server_key: "e7bcd6443f89a8f3b3f02528e1b5f5be"
server_key_iv: "23b520e93c8eff4faea1b82d953e729a"
server_key_version: "encryptionkey@0"
I20260929 18:42:10.116551 7911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "56b9c02a507d40de9340c99f8774c456"
format_stamp: "Formatted at 2026-09-29 18:42:10 on dist-test-slave-smpn"
server_key: "e7bcd6443f89a8f3b3f02528e1b5f5be"
server_key_iv: "23b520e93c8eff4faea1b82d953e729a"
server_key_version: "encryptionkey@0"
I20260929 18:42:10.120539 7911 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:42:10.123340 7926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:10.124795 7911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:42:10.124927 7911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "56b9c02a507d40de9340c99f8774c456"
format_stamp: "Formatted at 2026-09-29 18:42:10 on dist-test-slave-smpn"
server_key: "e7bcd6443f89a8f3b3f02528e1b5f5be"
server_key_iv: "23b520e93c8eff4faea1b82d953e729a"
server_key_version: "encryptionkey@0"
I20260929 18:42:10.125046 7911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:10.144948 7911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:10.148083 7911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:10.148280 7911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:10.148895 7911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:10.149753 7911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:10.149811 7911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:10.149864 7911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:10.149890 7911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:10.159582 7911 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:42007
I20260929 18:42:10.159626 8039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:42007 every 8 connection(s)
I20260929 18:42:10.160527 7911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:42:10.168457 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 7911
I20260929 18:42:10.168582 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:42:10.168885 7797 external_mini_cluster.cc:1428] Setting key cd96fc6e15a382d999da0f02cb9fdf94
Sep 29 18:42:10 dist-test-slave-smpn krb5kdc[7804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707330, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:10.175660 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:10.162421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:36681 (local address 127.7.157.126:40231)
0929 18:42:10.162598 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:10.162602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:10.163567 (+ 965us) server_negotiation.cc:407] Connection header received
0929 18:42:10.164955 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:10.164958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:10.165016 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:10.165105 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:10.166358 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:10.166923 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:10.168170 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:10.168523 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:10.168663 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:10.171521 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:10.171524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:10.171527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:10.171553 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:10.173243 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:10.173830 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:10.173834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:10.173836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:10.173931 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:10.174256 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:10.174259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:10.174260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:10.174509 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:10.174616 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:10.175303 (+ 687us) server_negotiation.cc:299] Negotiation successful
0929 18:42:10.175473 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 18:42:10.176494 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:10.162917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:40231 (local address 127.7.157.65:36681)
0929 18:42:10.163403 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:10.163450 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:10.164702 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:10.165269 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:10.165276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:10.165675 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:10.166188 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:10.166202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:10.167031 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:10.167034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:10.168009 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:10.168019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:10.168737 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:10.168741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:10.168896 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:10.169550 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:10.169582 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:10.171326 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:10.173382 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:10.173388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:10.173404 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:10.173703 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:10.174012 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:10.174014 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:10.174016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:10.174166 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:10.174653 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:10.174659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:10.174970 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 18:42:10.175208 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 18:42:10.175524 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":137,"threads_started":1}
I20260929 18:42:10.177886 8040 heartbeater.cc:347] Connected to a master server at 127.7.157.126:40231
I20260929 18:42:10.178130 8040 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
I20260929 18:42:10.178630 8040 heartbeater.cc:511] Master 127.7.157.126:40231 requested a full tablet report, sending...
I20260929 18:42:10.180166 7849 ts_manager.cc:194] Registered new tserver with Master: 56b9c02a507d40de9340c99f8774c456 (127.7.157.65:42007)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
I20260929 18:42:10.181332 7849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:36681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:42:10.197232 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:40231
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:10.312578 8045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:10.312872 8045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:10.312927 8045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:10.317103 8045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:10.317183 8045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:10.317258 8045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:42:10.322861 8045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:40231
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:10.324024 8045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:10.325088 8045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:10.335098 8050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:10.335132 8051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:10.335098 8053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:10.335281 8045 server_base.cc:1034] running on GCE node
I20260929 18:42:10.335820 8045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:10.336547 8045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:10.337710 8045 hybrid_clock.cc:648] HybridClock initialized: now 1790707330337688 us; error 33 us; skew 500 ppm
Sep 29 18:42:10 dist-test-slave-smpn krb5kdc[7804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707330, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:10.340516 8045 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:42:10.341444 8045 webserver.cc:460] Webserver started at http://127.7.157.66:42475/ using document root <none> and password file <none>
I20260929 18:42:10.342008 8045 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:10.342077 8045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:10.342268 8045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:10.344647 8045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "acf32f6cafbe43529f5c292b5e1b3c8f"
format_stamp: "Formatted at 2026-09-29 18:42:10 on dist-test-slave-smpn"
server_key: "24e5b7ebc57bfb21709bf038acf33ca9"
server_key_iv: "1d67abca626e365edba62acbffa53cca"
server_key_version: "encryptionkey@0"
I20260929 18:42:10.345312 8045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "acf32f6cafbe43529f5c292b5e1b3c8f"
format_stamp: "Formatted at 2026-09-29 18:42:10 on dist-test-slave-smpn"
server_key: "24e5b7ebc57bfb21709bf038acf33ca9"
server_key_iv: "1d67abca626e365edba62acbffa53cca"
server_key_version: "encryptionkey@0"
I20260929 18:42:10.349316 8045 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:42:10.351759 8060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:10.353016 8045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:10.353116 8045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "acf32f6cafbe43529f5c292b5e1b3c8f"
format_stamp: "Formatted at 2026-09-29 18:42:10 on dist-test-slave-smpn"
server_key: "24e5b7ebc57bfb21709bf038acf33ca9"
server_key_iv: "1d67abca626e365edba62acbffa53cca"
server_key_version: "encryptionkey@0"
I20260929 18:42:10.353225 8045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:10.366065 8045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:10.369437 8045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:10.369629 8045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:10.370237 8045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:10.371129 8045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:10.371176 8045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:10.371220 8045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:10.371249 8045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:10.379932 8045 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:32815
I20260929 18:42:10.379953 8173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:32815 every 8 connection(s)
I20260929 18:42:10.380858 8045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:42:10.383185 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 8045
I20260929 18:42:10.383296 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:42:10.383603 7797 external_mini_cluster.cc:1428] Setting key 0ecf9dc1ef51d10b5ab1da1286d91683
Sep 29 18:42:10 dist-test-slave-smpn krb5kdc[7804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707330, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
I20260929 18:42:10.395660 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:10.382692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:54247 (local address 127.7.157.126:40231)
0929 18:42:10.382813 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:10.382817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:10.383856 (+ 1039us) server_negotiation.cc:407] Connection header received
0929 18:42:10.385264 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:10.385266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:10.385312 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:10.385364 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:10.386913 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:10.387727 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:10.388996 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:10.389350 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:10.389458 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:10.392103 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:10.392108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:10.392112 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:10.392149 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:10.393629 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:10.394100 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:10.394102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:10.394103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:10.394173 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:10.394579 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:10.394581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:10.394582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:10.394758 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:10.394830 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:10.395410 (+ 580us) server_negotiation.cc:299] Negotiation successful
0929 18:42:10.395515 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
I20260929 18:42:10.396674 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:10.383118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:40231 (local address 127.7.157.66:54247)
0929 18:42:10.383665 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:10.383731 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:10.385026 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:10.385571 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:10.385581 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:10.386052 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:10.386755 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:10.386777 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:10.387892 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:10.387895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:10.388857 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:10.388869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:10.389507 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:10.389510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:10.389668 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:10.390265 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:10.390282 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:10.391896 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:10.393728 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:10.393734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:10.393747 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:10.394007 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:10.394283 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:10.394287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:10.394288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:10.394483 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:10.394855 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:10.394861 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:10.395172 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 18:42:10.395379 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 18:42:10.395680 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":128,"threads_started":1}
I20260929 18:42:10.397982 8174 heartbeater.cc:347] Connected to a master server at 127.7.157.126:40231
I20260929 18:42:10.398213 8174 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:10.398789 8174 heartbeater.cc:511] Master 127.7.157.126:40231 requested a full tablet report, sending...
I20260929 18:42:10.399950 7849 ts_manager.cc:194] Registered new tserver with Master: acf32f6cafbe43529f5c292b5e1b3c8f (127.7.157.66:32815)
I20260929 18:42:10.400732 7849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:54247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:42:10.411427 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:40231
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:10.524997 8179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:10.525264 8179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:10.525307 8179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:10.529403 8179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:10.529465 8179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:10.529531 8179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:42:10.534946 8179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:40231
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:10.535974 8179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:10.536972 8179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:10.546316 8184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:10.546375 8187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:10.546442 8179 server_base.cc:1034] running on GCE node
W20260929 18:42:10.546659 8185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:10.547129 8179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:10.547842 8179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:10.548996 8179 hybrid_clock.cc:648] HybridClock initialized: now 1790707330548971 us; error 40 us; skew 500 ppm
Sep 29 18:42:10 dist-test-slave-smpn krb5kdc[7804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707330, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:10.551687 8179 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:42:10.552549 8179 webserver.cc:460] Webserver started at http://127.7.157.67:35721/ using document root <none> and password file <none>
I20260929 18:42:10.553076 8179 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:10.553133 8179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:10.553303 8179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:10.555433 8179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "c4d5764851274c95b39968dcbaaa24f4"
format_stamp: "Formatted at 2026-09-29 18:42:10 on dist-test-slave-smpn"
server_key: "d0283d85b89c485db6f39777577c3378"
server_key_iv: "0634c67995a9e4313a532eeae7e219ec"
server_key_version: "encryptionkey@0"
I20260929 18:42:10.555943 8179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "c4d5764851274c95b39968dcbaaa24f4"
format_stamp: "Formatted at 2026-09-29 18:42:10 on dist-test-slave-smpn"
server_key: "d0283d85b89c485db6f39777577c3378"
server_key_iv: "0634c67995a9e4313a532eeae7e219ec"
server_key_version: "encryptionkey@0"
I20260929 18:42:10.559530 8179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:42:10.561863 8194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:10.563092 8179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:10.563190 8179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "c4d5764851274c95b39968dcbaaa24f4"
format_stamp: "Formatted at 2026-09-29 18:42:10 on dist-test-slave-smpn"
server_key: "d0283d85b89c485db6f39777577c3378"
server_key_iv: "0634c67995a9e4313a532eeae7e219ec"
server_key_version: "encryptionkey@0"
I20260929 18:42:10.563275 8179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:10.589241 8179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:10.592031 8179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:10.592183 8179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:10.593127 8179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:10.593952 8179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:10.593997 8179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:10.594034 8179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:10.594054 8179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:10.602720 8179 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:35131
I20260929 18:42:10.602762 8307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:35131 every 8 connection(s)
I20260929 18:42:10.603693 8179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:42:10.607919 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 8179
I20260929 18:42:10.608028 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:42:10.608456 7797 external_mini_cluster.cc:1428] Setting key fa0217af92b662779cd9bd5d7d561952
Sep 29 18:42:10 dist-test-slave-smpn krb5kdc[7804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707330, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:10.617288 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:10.605583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:38773 (local address 127.7.157.126:40231)
0929 18:42:10.605714 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:10.605718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:10.606568 (+ 850us) server_negotiation.cc:407] Connection header received
0929 18:42:10.608057 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:10.608060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:10.608131 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:10.608216 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:10.609649 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:10.610260 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:10.611685 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:10.611943 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:10.612022 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:10.614359 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:10.614362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:10.614364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:10.614389 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:10.615675 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:10.616096 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:10.616098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:10.616100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:10.616168 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:10.616411 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:10.616413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:10.616414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:10.616575 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:10.616641 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:10.617058 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 18:42:10.617164 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:42:10.618095 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:10.605982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:40231 (local address 127.7.157.67:38773)
0929 18:42:10.606426 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:10.606470 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:10.607793 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:10.608429 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:10.608441 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:10.608943 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:10.609525 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:10.609548 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:10.610376 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:10.610380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:10.611580 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:10.611589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:10.612062 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:10.612064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:10.612214 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:10.612744 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:10.612759 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:10.614205 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:10.615755 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:10.615760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:10.615775 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:10.616016 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:10.616227 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:10.616229 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:10.616230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:10.616352 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:10.616642 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:10.616646 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:10.616875 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 18:42:10.617027 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 18:42:10.617276 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20260929 18:42:10.619161 8308 heartbeater.cc:347] Connected to a master server at 127.7.157.126:40231
I20260929 18:42:10.619391 8308 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:10.619863 8308 heartbeater.cc:511] Master 127.7.157.126:40231 requested a full tablet report, sending...
I20260929 18:42:10.620831 7849 ts_manager.cc:194] Registered new tserver with Master: c4d5764851274c95b39968dcbaaa24f4 (127.7.157.67:35131)
I20260929 18:42:10.621395 7849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:38773
I20260929 18:42:10.622900 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:42:10 dist-test-slave-smpn krb5kdc[7804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:42:10 dist-test-slave-smpn krb5kdc[7804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.157.65@KRBTEST.COM
I20260929 18:42:10.646883 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:10.638080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34578 (local address 127.7.157.65:42007)
0929 18:42:10.638353 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:10.638368 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:10.638396 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:42:10.638538 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:10.638544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:10.638748 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:10.638883 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:10.639632 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:10.640409 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:10.641248 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:10.641619 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:10.641714 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:10.644006 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:10.644015 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:10.644030 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:10.644057 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:10.645438 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:10.645814 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:10.645819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:10.645820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:10.645894 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:10.646082 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:10.646084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:10.646085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:10.646315 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:10.646485 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:10.646610 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 18:42:10.646732 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
W20260929 18:42:10.647874 7954 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34578
Sep 29 18:42:10 dist-test-slave-smpn krb5kdc[7804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:42:10 dist-test-slave-smpn krb5kdc[7804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.65@KRBTEST.COM
I20260929 18:42:10.671797 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:10.664309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.7.157.65:42007)
0929 18:42:10.664451 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:10.664456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:10.664558 (+ 102us) server_negotiation.cc:407] Connection header received
0929 18:42:10.664709 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:10.664712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:10.664758 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:10.664825 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:10.665480 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:10.665983 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:10.666859 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:10.667130 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:10.667224 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:10.669369 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:10.669373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:10.669375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:10.669400 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:10.670478 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:10.670829 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:10.670831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:10.670832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:10.670899 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:10.671148 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:10.671150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:10.671151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:10.671304 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:10.671397 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:10.671581 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 18:42:10.671677 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:42:10.672729 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 7911
I20260929 18:42:10.677461 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 8045
I20260929 18:42:10.682037 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 8179
I20260929 18:42:10.686677 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 7817
2026-09-29T18:42:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2060 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:42:10 dist-test-slave-smpn krb5kdc[8322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:42:10 dist-test-slave-smpn krb5kdc[8322](info): set up 2 sockets
Sep 29 18:42:10 dist-test-slave-smpn krb5kdc[8322](info): commencing operation
krb5kdc: starting...
W20260929 18:42:11.634502 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:42:11 dist-test-slave-smpn krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:42:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:42:11.750043 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:35847
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:35847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:11.863859 8335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:11.864145 8335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:11.864194 8335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:11.868302 8335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:11.868381 8335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:11.868399 8335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:11.868415 8335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:11.868439 8335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:11.874028 8335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:35847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:35847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:11.875212 8335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:11.876283 8335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:42:11.884838 8335 server_base.cc:1034] running on GCE node
W20260929 18:42:11.884826 8340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:11.884840 8341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:11.884805 8343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:11.885627 8335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:11.886741 8335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:11.887895 8335 hybrid_clock.cc:648] HybridClock initialized: now 1790707331887877 us; error 28 us; skew 500 ppm
Sep 29 18:42:11 dist-test-slave-smpn krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707331, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:11.890568 8335 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:42:11.891499 8335 webserver.cc:460] Webserver started at http://127.7.157.126:38267/ using document root <none> and password file <none>
I20260929 18:42:11.892036 8335 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:11.892110 8335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:11.892320 8335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:11.894410 8335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "037f6dba325a40dfacd961754eb5c4c0"
format_stamp: "Formatted at 2026-09-29 18:42:11 on dist-test-slave-smpn"
server_key: "4e11a67b99149fbe231084974ea6b88a"
server_key_iv: "89c935d469e621c25dcea5842677a07d"
server_key_version: "encryptionkey@0"
I20260929 18:42:11.894974 8335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "037f6dba325a40dfacd961754eb5c4c0"
format_stamp: "Formatted at 2026-09-29 18:42:11 on dist-test-slave-smpn"
server_key: "4e11a67b99149fbe231084974ea6b88a"
server_key_iv: "89c935d469e621c25dcea5842677a07d"
server_key_version: "encryptionkey@0"
I20260929 18:42:11.898754 8335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:42:11.901252 8350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:11.902544 8335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:11.902675 8335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "037f6dba325a40dfacd961754eb5c4c0"
format_stamp: "Formatted at 2026-09-29 18:42:11 on dist-test-slave-smpn"
server_key: "4e11a67b99149fbe231084974ea6b88a"
server_key_iv: "89c935d469e621c25dcea5842677a07d"
server_key_version: "encryptionkey@0"
I20260929 18:42:11.902786 8335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:11.927167 8335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:11.930325 8335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:11.930501 8335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:11.937354 8335 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:35847
I20260929 18:42:11.937384 8402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:35847 every 8 connection(s)
I20260929 18:42:11.938340 8335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:11.941167 8403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:11.945981 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 8335
I20260929 18:42:11.946105 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:11.946368 7797 external_mini_cluster.cc:1428] Setting key 643b8c51b33eb594093aaebd648c92a0
I20260929 18:42:11.947113 8403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0: Bootstrap starting.
I20260929 18:42:11.949508 8403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:11.950500 8403 log.cc:824] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:11.952657 8403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0: No bootstrap required, opened a new log
Sep 29 18:42:11 dist-test-slave-smpn krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:11.956802 8403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "037f6dba325a40dfacd961754eb5c4c0" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35847 } }
I20260929 18:42:11.957050 8403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:11.957101 8403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 037f6dba325a40dfacd961754eb5c4c0, State: Initialized, Role: FOLLOWER
I20260929 18:42:11.957566 8403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "037f6dba325a40dfacd961754eb5c4c0" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35847 } }
I20260929 18:42:11.957702 8403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:11.957767 8403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:11.957862 8403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:11.958755 8403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "037f6dba325a40dfacd961754eb5c4c0" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35847 } }
I20260929 18:42:11.959170 8403 leader_election.cc:302] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 037f6dba325a40dfacd961754eb5c4c0; no voters:
I20260929 18:42:11.959568 8403 leader_election.cc:288] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:11.959651 8408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:11.959823 8408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 037f6dba325a40dfacd961754eb5c4c0, State: Running, Role: LEADER
I20260929 18:42:11.960151 8408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "037f6dba325a40dfacd961754eb5c4c0" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35847 } }
I20260929 18:42:11.960711 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:11.947779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.7.157.126:35847)
0929 18:42:11.948182 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:11.948200 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:11.948235 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:42:11.948880 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:11.948898 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:11.949260 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:11.949564 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:11.950443 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:11.951515 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:11.952603 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:11.953099 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:11.953212 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:11.955864 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:11.955869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:11.955884 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:11.955911 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:11.957598 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:11.958052 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:11.958057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:11.958062 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:11.958177 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:11.958428 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:11.958431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:11.958433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:11.958843 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:11.959009 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:11.959308 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 18:42:11.959641 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
I20260929 18:42:11.961330 8403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:11.962025 8409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "037f6dba325a40dfacd961754eb5c4c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "037f6dba325a40dfacd961754eb5c4c0" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35847 } } }
I20260929 18:42:11.962066 8410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 037f6dba325a40dfacd961754eb5c4c0. Latest consensus state: current_term: 1 leader_uuid: "037f6dba325a40dfacd961754eb5c4c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "037f6dba325a40dfacd961754eb5c4c0" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35847 } } }
I20260929 18:42:11.962164 8409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:11.962184 8410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0 [sys.catalog]: This master's current role is: LEADER
W20260929 18:42:11.965935 8423 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:42:11.966008 8423 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:42:11.966066 8415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:11.966724 8415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:11.972497 8415 catalog_manager.cc:1347] Generated new cluster ID: 4ddd9ad500b14309977e1260f9478604
I20260929 18:42:11.972585 8415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:11.989648 8415 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:11.990654 8415 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:12.002898 8415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 037f6dba325a40dfacd961754eb5c4c0: Generated new TSK 0
I20260929 18:42:12.003908 8415 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:42:12.037959 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35847
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:12.152599 8429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:12.152900 8429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:12.152945 8429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:12.157166 8429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:12.157246 8429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:12.157327 8429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:42:12.163004 8429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35847
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:12.164196 8429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:12.165222 8429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:42:12.175141 8429 server_base.cc:1034] running on GCE node
W20260929 18:42:12.175030 8434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:12.175030 8437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:12.175033 8435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:12.175721 8429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:12.176406 8429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:12.177572 8429 hybrid_clock.cc:648] HybridClock initialized: now 1790707332177555 us; error 37 us; skew 500 ppm
Sep 29 18:42:12 dist-test-slave-smpn krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707332, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:12.180366 8429 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:42:12.181310 8429 webserver.cc:460] Webserver started at http://127.7.157.65:41563/ using document root <none> and password file <none>
I20260929 18:42:12.181871 8429 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:12.181936 8429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:12.182163 8429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:12.184347 8429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "734f68f73bb74817b3eb1124a43fc30e"
format_stamp: "Formatted at 2026-09-29 18:42:12 on dist-test-slave-smpn"
server_key: "74a4eb5733143ef474fef1134074b6be"
server_key_iv: "3e0397b39d77764ff6c6a8c4648c77f7"
server_key_version: "encryptionkey@0"
I20260929 18:42:12.184894 8429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "734f68f73bb74817b3eb1124a43fc30e"
format_stamp: "Formatted at 2026-09-29 18:42:12 on dist-test-slave-smpn"
server_key: "74a4eb5733143ef474fef1134074b6be"
server_key_iv: "3e0397b39d77764ff6c6a8c4648c77f7"
server_key_version: "encryptionkey@0"
I20260929 18:42:12.188668 8429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:42:12.191263 8444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:12.192605 8429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:42:12.192720 8429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "734f68f73bb74817b3eb1124a43fc30e"
format_stamp: "Formatted at 2026-09-29 18:42:12 on dist-test-slave-smpn"
server_key: "74a4eb5733143ef474fef1134074b6be"
server_key_iv: "3e0397b39d77764ff6c6a8c4648c77f7"
server_key_version: "encryptionkey@0"
I20260929 18:42:12.192848 8429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:12.205340 8429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:12.208642 8429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:12.208812 8429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:12.209455 8429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:12.210305 8429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:12.210350 8429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:12.210391 8429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:12.210419 8429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:12.218921 8429 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:46629
I20260929 18:42:12.218943 8557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:46629 every 8 connection(s)
I20260929 18:42:12.219909 8429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:42:12.224303 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 8429
I20260929 18:42:12.224406 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:42:12.224653 7797 external_mini_cluster.cc:1428] Setting key 5e8ec17d193e14de5ed4db396a5e9c94
Sep 29 18:42:12 dist-test-slave-smpn krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707332, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
I20260929 18:42:12.234186 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:12.221665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:58707 (local address 127.7.157.126:35847)
0929 18:42:12.221834 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:12.221838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:12.222771 (+ 933us) server_negotiation.cc:407] Connection header received
0929 18:42:12.224265 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:12.224269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:12.224330 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:12.224407 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:12.225744 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.226360 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.227703 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.227945 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.228032 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:12.230582 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:12.230585 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:12.230587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:12.230610 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:12.231931 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:12.232515 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:12.232517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:12.232519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:12.232590 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:12.232989 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:12.232992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:12.232993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:12.233197 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:12.233288 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:12.233939 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 18:42:12.234060 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 18:42:12.235013 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:12.222139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:35847 (local address 127.7.157.65:58707)
0929 18:42:12.222602 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:12.222648 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:12.224014 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:12.224620 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:12.224629 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:12.225037 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:12.225604 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:12.225618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.226480 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.226483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:12.227565 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:12.227587 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.228095 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.228099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:12.228302 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:12.228910 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:12.228927 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:12.230408 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:12.232047 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:12.232054 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:12.232075 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:12.232415 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:12.232669 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:12.232672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:12.232674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:12.232861 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:12.233308 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:12.233315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:12.233675 (+ 360us) client_negotiation.cc:769] Sending connection context
0929 18:42:12.233874 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 18:42:12.234148 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
I20260929 18:42:12.236335 8558 heartbeater.cc:347] Connected to a master server at 127.7.157.126:35847
I20260929 18:42:12.236585 8558 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:12.237068 8558 heartbeater.cc:511] Master 127.7.157.126:35847 requested a full tablet report, sending...
I20260929 18:42:12.238649 8367 ts_manager.cc:194] Registered new tserver with Master: 734f68f73bb74817b3eb1124a43fc30e (127.7.157.65:46629)
I20260929 18:42:12.239853 8367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:58707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:42:12.249796 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35847
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:12.364218 8563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:12.364531 8563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:12.364609 8563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:12.368891 8563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:12.368973 8563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:12.369057 8563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:42:12.374804 8563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35847
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:12.376008 8563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:12.377058 8563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:12.386819 8569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:12.386804 8571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:12.386804 8568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:12.386870 8563 server_base.cc:1034] running on GCE node
I20260929 18:42:12.387475 8563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:12.388136 8563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:12.389313 8563 hybrid_clock.cc:648] HybridClock initialized: now 1790707332389295 us; error 27 us; skew 500 ppm
Sep 29 18:42:12 dist-test-slave-smpn krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707332, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:12.391969 8563 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:42:12.392839 8563 webserver.cc:460] Webserver started at http://127.7.157.66:41363/ using document root <none> and password file <none>
I20260929 18:42:12.393366 8563 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:12.393422 8563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:12.393589 8563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:12.395704 8563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "60aedfc7473b494aa0fe8224a34ddc6b"
format_stamp: "Formatted at 2026-09-29 18:42:12 on dist-test-slave-smpn"
server_key: "ec214fc15acada96ae7dcdf4f94e1f24"
server_key_iv: "a9b8730609bea4c566d676d1ab00dddb"
server_key_version: "encryptionkey@0"
I20260929 18:42:12.396286 8563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "60aedfc7473b494aa0fe8224a34ddc6b"
format_stamp: "Formatted at 2026-09-29 18:42:12 on dist-test-slave-smpn"
server_key: "ec214fc15acada96ae7dcdf4f94e1f24"
server_key_iv: "a9b8730609bea4c566d676d1ab00dddb"
server_key_version: "encryptionkey@0"
I20260929 18:42:12.400166 8563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:42:12.402724 8578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:12.404033 8563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:12.404170 8563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "60aedfc7473b494aa0fe8224a34ddc6b"
format_stamp: "Formatted at 2026-09-29 18:42:12 on dist-test-slave-smpn"
server_key: "ec214fc15acada96ae7dcdf4f94e1f24"
server_key_iv: "a9b8730609bea4c566d676d1ab00dddb"
server_key_version: "encryptionkey@0"
I20260929 18:42:12.404274 8563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:12.424588 8563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:12.427529 8563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:12.427716 8563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:12.428315 8563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:12.429152 8563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:12.429198 8563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:12.429250 8563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:12.429277 8563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:12.437827 8563 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:44553
I20260929 18:42:12.437848 8691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:44553 every 8 connection(s)
I20260929 18:42:12.438840 8563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:42:12.446632 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 8563
I20260929 18:42:12.446746 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:42:12.446987 7797 external_mini_cluster.cc:1428] Setting key c60b65eb70e0f0bc8457e7ded364350e
Sep 29 18:42:12 dist-test-slave-smpn krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707332, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:12.452666 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:12.440721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:41153 (local address 127.7.157.126:35847)
0929 18:42:12.440840 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:12.440844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:12.441678 (+ 834us) server_negotiation.cc:407] Connection header received
0929 18:42:12.443025 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:12.443028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:12.443110 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:12.443187 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:12.444420 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.444953 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.446087 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.446310 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.446419 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:12.449136 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:12.449139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:12.449141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:12.449164 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:12.450593 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:12.451102 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:12.451105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:12.451107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:12.451194 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:12.451555 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:12.451557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:12.451559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:12.451767 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:12.451850 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:12.452351 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 18:42:12.452481 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:42:12.453493 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:12.441084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:35847 (local address 127.7.157.66:41153)
0929 18:42:12.441519 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:12.441569 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:12.442804 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:12.443375 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:12.443382 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:12.443743 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:12.444271 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:12.444286 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.445080 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.445083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:12.445970 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:12.445978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.446449 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.446453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:12.446623 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:12.447226 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:12.447245 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:12.448953 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:12.450714 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:12.450719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:12.450733 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:12.450988 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:12.451273 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:12.451277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:12.451279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:12.451466 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:12.451856 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:12.451862 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:12.452119 (+ 257us) client_negotiation.cc:769] Sending connection context
0929 18:42:12.452279 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 18:42:12.452609 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":102,"threads_started":1}
I20260929 18:42:12.454695 8692 heartbeater.cc:347] Connected to a master server at 127.7.157.126:35847
I20260929 18:42:12.454975 8692 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:12.455641 8692 heartbeater.cc:511] Master 127.7.157.126:35847 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
I20260929 18:42:12.456766 8367 ts_manager.cc:194] Registered new tserver with Master: 60aedfc7473b494aa0fe8224a34ddc6b (127.7.157.66:44553)
I20260929 18:42:12.457376 8367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:41153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:42:12.473429 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35847
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:12.582484 8697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:12.582832 8697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:12.582901 8697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:12.587107 8697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:12.587172 8697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:12.587241 8697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:42:12.592499 8697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35847
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:12.593500 8697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:12.594465 8697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:12.604180 8703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:12.604203 8697 server_base.cc:1034] running on GCE node
W20260929 18:42:12.604279 8702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:12.604192 8705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:12.604837 8697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:12.605460 8697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:12.606601 8697 hybrid_clock.cc:648] HybridClock initialized: now 1790707332606580 us; error 49 us; skew 500 ppm
Sep 29 18:42:12 dist-test-slave-smpn krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707332, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:12.609333 8697 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:42:12.610193 8697 webserver.cc:460] Webserver started at http://127.7.157.67:43961/ using document root <none> and password file <none>
I20260929 18:42:12.610737 8697 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:12.610813 8697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:12.611017 8697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:12.613170 8697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff"
format_stamp: "Formatted at 2026-09-29 18:42:12 on dist-test-slave-smpn"
server_key: "b6544db73e05deb2226520b8796ba27a"
server_key_iv: "927964e2817864c7b6c9abf60c807f69"
server_key_version: "encryptionkey@0"
I20260929 18:42:12.613725 8697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff"
format_stamp: "Formatted at 2026-09-29 18:42:12 on dist-test-slave-smpn"
server_key: "b6544db73e05deb2226520b8796ba27a"
server_key_iv: "927964e2817864c7b6c9abf60c807f69"
server_key_version: "encryptionkey@0"
I20260929 18:42:12.617520 8697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:42:12.620016 8712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:12.621282 8697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:12.621403 8697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff"
format_stamp: "Formatted at 2026-09-29 18:42:12 on dist-test-slave-smpn"
server_key: "b6544db73e05deb2226520b8796ba27a"
server_key_iv: "927964e2817864c7b6c9abf60c807f69"
server_key_version: "encryptionkey@0"
I20260929 18:42:12.621506 8697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:12.635424 8697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:12.638584 8697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:12.638767 8697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:12.639411 8697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:12.640254 8697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:12.640303 8697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:12.640353 8697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:12.640383 8697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:12.649047 8697 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:37379
I20260929 18:42:12.649081 8825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:37379 every 8 connection(s)
I20260929 18:42:12.650051 8697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:42:12.659271 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 8697
Sep 29 18:42:12 dist-test-slave-smpn krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707332, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:12.659390 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:42:12.659698 7797 external_mini_cluster.cc:1428] Setting key 9c7e679d142ff498084f0a9253418850
I20260929 18:42:12.663841 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:12.651925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:51327 (local address 127.7.157.126:35847)
0929 18:42:12.652052 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:12.652056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:12.652908 (+ 852us) server_negotiation.cc:407] Connection header received
0929 18:42:12.654218 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:12.654222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:12.654277 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:12.654350 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:12.655580 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.656102 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.657238 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.657458 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.657548 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:12.660136 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:12.660140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:12.660143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:12.660173 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:12.661726 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:12.662280 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:12.662283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:12.662285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:12.662370 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:12.662701 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:12.662704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:12.662706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:12.662902 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:12.662987 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:12.663557 (+ 570us) server_negotiation.cc:299] Negotiation successful
0929 18:42:12.663681 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:42:12.664745 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:12.652332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:35847 (local address 127.7.157.67:51327)
0929 18:42:12.652757 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:12.652799 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:12.653997 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:12.654532 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:12.654540 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:12.654902 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:12.655437 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:12.655451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.656218 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.656220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:12.657118 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:12.657128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.657618 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.657622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:12.657781 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:12.658306 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:12.658323 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:12.659931 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:12.661841 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:12.661847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:12.661866 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:12.662190 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:12.662450 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:12.662453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:12.662455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:12.662617 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:12.663000 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:12.663008 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:12.663323 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 18:42:12.663505 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 18:42:12.663874 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":97,"threads_started":1}
I20260929 18:42:12.665846 8826 heartbeater.cc:347] Connected to a master server at 127.7.157.126:35847
I20260929 18:42:12.666085 8826 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:12.666611 8826 heartbeater.cc:511] Master 127.7.157.126:35847 requested a full tablet report, sending...
I20260929 18:42:12.667672 8367 ts_manager.cc:194] Registered new tserver with Master: d5ddf2e9c39d43a09ba7fe8b778598ff (127.7.157.67:37379)
I20260929 18:42:12.668219 8367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:51327
I20260929 18:42:12.673686 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:42:12.684109 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:12.677166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.7.157.126:35847)
0929 18:42:12.677312 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:12.677316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:12.677440 (+ 124us) server_negotiation.cc:407] Connection header received
0929 18:42:12.677569 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:12.677572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:12.677621 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:12.677690 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:12.678380 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.678898 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.679753 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.679980 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.680069 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:12.681340 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:12.681344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:12.681347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:12.681373 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:12.682513 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:12.682895 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:12.682898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:12.682900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:12.682982 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:12.683253 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:12.683256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:12.683258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:12.683480 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:12.683598 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:12.683828 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 18:42:12.683943 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:42:12.688853 8367 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:42:12.692023 8367 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:12.706408 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:12.701172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33960 (local address 127.7.157.65:46629)
0929 18:42:12.701579 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:12.701594 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:12.701798 (+ 204us) server_negotiation.cc:407] Connection header received
0929 18:42:12.701987 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:12.701993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:12.702179 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:12.702287 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:12.702651 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.703421 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.705157 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.705726 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.705828 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:12.705985 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:12.706171 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 18:42:12.706268 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":72,"threads_started":1}
I20260929 18:42:12.706403 8837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:12.700708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:46629 (local address 127.0.0.1:33960)
0929 18:42:12.701720 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:12.701732 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:12.701859 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:12.702439 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:12.702442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:12.702459 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:12.702543 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:12.702549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.703536 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.703540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:12.705025 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:12.705032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.705828 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.705832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:12.705969 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:12.706008 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 18:42:12.706126 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 18:42:12.706251 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"mutex_wait_us":58,"thread_start_us":92,"threads_started":1}
I20260929 18:42:12.706732 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:12.700504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:44553 (local address 127.0.0.1:51944)
0929 18:42:12.701072 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:12.701126 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:12.701246 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:12.702117 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:12.702124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:12.702147 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:12.702319 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:12.702326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.703795 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.703798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:12.704993 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:12.705004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.706266 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.706270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:12.706386 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:12.706422 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 18:42:12.706487 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 18:42:12.706568 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":71,"thread_start_us":106,"threads_started":1}
I20260929 18:42:12.706794 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:12.700505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.7.157.67:37379)
0929 18:42:12.701506 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:12.701518 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:12.701539 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:42:12.701631 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:12.701636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:12.701838 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:12.701982 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:12.702688 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.703814 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.705159 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.705955 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.706071 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:12.706203 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:12.706559 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 18:42:12.706658 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":136,"threads_started":1}
I20260929 18:42:12.707099 8842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:12.701172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.7.157.66:44553)
0929 18:42:12.701651 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:12.701661 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:12.701682 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:42:12.701757 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:12.701762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:12.701946 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:12.702086 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:12.702438 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.703671 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.705124 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.705955 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.706056 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:12.706203 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:12.706831 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 18:42:12.706924 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":87,"threads_started":1}
I20260929 18:42:12.707132 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:12.700390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:37379 (local address 127.0.0.1:34614)
0929 18:42:12.701072 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:12.701253 (+ 181us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:12.701321 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:12.702157 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:12.702159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:12.702172 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:12.702319 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:12.702326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.703939 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.703942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:12.705033 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:12.705042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:12.706266 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:12.706270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:12.706413 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:12.706431 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:42:12.706487 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:42:12.706997 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":564,"thread_start_us":86,"threads_started":1}
I20260929 18:42:12.709820 8760 tablet_service.cc:1467] Processing CreateTablet for tablet ca08d1cac1d3490c8aa7e0ec15d0966a (DEFAULT_TABLE table=test-table [id=ce5eaed5911e44e5b468e8ece0ea71ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:12.709821 8492 tablet_service.cc:1467] Processing CreateTablet for tablet ca08d1cac1d3490c8aa7e0ec15d0966a (DEFAULT_TABLE table=test-table [id=ce5eaed5911e44e5b468e8ece0ea71ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:12.710776 8760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca08d1cac1d3490c8aa7e0ec15d0966a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:12.710777 8492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca08d1cac1d3490c8aa7e0ec15d0966a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:12.716961 8843 tablet_bootstrap.cc:492] T ca08d1cac1d3490c8aa7e0ec15d0966a P 734f68f73bb74817b3eb1124a43fc30e: Bootstrap starting.
I20260929 18:42:12.717211 8626 tablet_service.cc:1467] Processing CreateTablet for tablet ca08d1cac1d3490c8aa7e0ec15d0966a (DEFAULT_TABLE table=test-table [id=ce5eaed5911e44e5b468e8ece0ea71ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:12.717782 8844 tablet_bootstrap.cc:492] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff: Bootstrap starting.
I20260929 18:42:12.718324 8626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca08d1cac1d3490c8aa7e0ec15d0966a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:12.719678 8843 tablet_bootstrap.cc:654] T ca08d1cac1d3490c8aa7e0ec15d0966a P 734f68f73bb74817b3eb1124a43fc30e: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:12.720479 8844 tablet_bootstrap.cc:654] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:12.720945 8843 log.cc:824] T ca08d1cac1d3490c8aa7e0ec15d0966a P 734f68f73bb74817b3eb1124a43fc30e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:12.721290 8844 log.cc:824] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:12.722820 8843 tablet_bootstrap.cc:492] T ca08d1cac1d3490c8aa7e0ec15d0966a P 734f68f73bb74817b3eb1124a43fc30e: No bootstrap required, opened a new log
I20260929 18:42:12.723017 8843 ts_tablet_manager.cc:1397] T ca08d1cac1d3490c8aa7e0ec15d0966a P 734f68f73bb74817b3eb1124a43fc30e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:42:12.723327 8844 tablet_bootstrap.cc:492] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff: No bootstrap required, opened a new log
I20260929 18:42:12.723531 8844 ts_tablet_manager.cc:1397] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:42:12.726228 8843 raft_consensus.cc:348] T ca08d1cac1d3490c8aa7e0ec15d0966a P 734f68f73bb74817b3eb1124a43fc30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37379 } } peers { permanent_uuid: "60aedfc7473b494aa0fe8224a34ddc6b" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 44553 } } peers { permanent_uuid: "734f68f73bb74817b3eb1124a43fc30e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 46629 } }
I20260929 18:42:12.726522 8843 raft_consensus.cc:374] T ca08d1cac1d3490c8aa7e0ec15d0966a P 734f68f73bb74817b3eb1124a43fc30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:12.726588 8843 raft_consensus.cc:729] T ca08d1cac1d3490c8aa7e0ec15d0966a P 734f68f73bb74817b3eb1124a43fc30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734f68f73bb74817b3eb1124a43fc30e, State: Initialized, Role: FOLLOWER
I20260929 18:42:12.726935 8847 tablet_bootstrap.cc:492] T ca08d1cac1d3490c8aa7e0ec15d0966a P 60aedfc7473b494aa0fe8224a34ddc6b: Bootstrap starting.
I20260929 18:42:12.727020 8843 consensus_queue.cc:260] T ca08d1cac1d3490c8aa7e0ec15d0966a P 734f68f73bb74817b3eb1124a43fc30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37379 } } peers { permanent_uuid: "60aedfc7473b494aa0fe8224a34ddc6b" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 44553 } } peers { permanent_uuid: "734f68f73bb74817b3eb1124a43fc30e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 46629 } }
I20260929 18:42:12.727257 8844 raft_consensus.cc:348] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37379 } } peers { permanent_uuid: "60aedfc7473b494aa0fe8224a34ddc6b" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 44553 } } peers { permanent_uuid: "734f68f73bb74817b3eb1124a43fc30e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 46629 } }
I20260929 18:42:12.727535 8844 raft_consensus.cc:374] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:12.727593 8844 raft_consensus.cc:729] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ddf2e9c39d43a09ba7fe8b778598ff, State: Initialized, Role: FOLLOWER
I20260929 18:42:12.727823 8558 heartbeater.cc:503] Master 127.7.157.126:35847 was elected leader, sending a full tablet report...
I20260929 18:42:12.728075 8843 ts_tablet_manager.cc:1428] T ca08d1cac1d3490c8aa7e0ec15d0966a P 734f68f73bb74817b3eb1124a43fc30e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:42:12.728063 8844 consensus_queue.cc:260] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37379 } } peers { permanent_uuid: "60aedfc7473b494aa0fe8224a34ddc6b" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 44553 } } peers { permanent_uuid: "734f68f73bb74817b3eb1124a43fc30e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 46629 } }
I20260929 18:42:12.728770 8826 heartbeater.cc:503] Master 127.7.157.126:35847 was elected leader, sending a full tablet report...
I20260929 18:42:12.729009 8844 ts_tablet_manager.cc:1428] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:42:12.729700 8847 tablet_bootstrap.cc:654] T ca08d1cac1d3490c8aa7e0ec15d0966a P 60aedfc7473b494aa0fe8224a34ddc6b: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:12.730573 8847 log.cc:824] T ca08d1cac1d3490c8aa7e0ec15d0966a P 60aedfc7473b494aa0fe8224a34ddc6b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:12.732944 8847 tablet_bootstrap.cc:492] T ca08d1cac1d3490c8aa7e0ec15d0966a P 60aedfc7473b494aa0fe8224a34ddc6b: No bootstrap required, opened a new log
I20260929 18:42:12.733137 8847 ts_tablet_manager.cc:1397] T ca08d1cac1d3490c8aa7e0ec15d0966a P 60aedfc7473b494aa0fe8224a34ddc6b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:42:12.736567 8847 raft_consensus.cc:348] T ca08d1cac1d3490c8aa7e0ec15d0966a P 60aedfc7473b494aa0fe8224a34ddc6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37379 } } peers { permanent_uuid: "60aedfc7473b494aa0fe8224a34ddc6b" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 44553 } } peers { permanent_uuid: "734f68f73bb74817b3eb1124a43fc30e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 46629 } }
I20260929 18:42:12.736819 8847 raft_consensus.cc:374] T ca08d1cac1d3490c8aa7e0ec15d0966a P 60aedfc7473b494aa0fe8224a34ddc6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:12.736879 8847 raft_consensus.cc:729] T ca08d1cac1d3490c8aa7e0ec15d0966a P 60aedfc7473b494aa0fe8224a34ddc6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60aedfc7473b494aa0fe8224a34ddc6b, State: Initialized, Role: FOLLOWER
I20260929 18:42:12.737294 8847 consensus_queue.cc:260] T ca08d1cac1d3490c8aa7e0ec15d0966a P 60aedfc7473b494aa0fe8224a34ddc6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37379 } } peers { permanent_uuid: "60aedfc7473b494aa0fe8224a34ddc6b" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 44553 } } peers { permanent_uuid: "734f68f73bb74817b3eb1124a43fc30e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 46629 } }
I20260929 18:42:12.737948 8692 heartbeater.cc:503] Master 127.7.157.126:35847 was elected leader, sending a full tablet report...
I20260929 18:42:12.738286 8847 ts_tablet_manager.cc:1428] T ca08d1cac1d3490c8aa7e0ec15d0966a P 60aedfc7473b494aa0fe8224a34ddc6b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260929 18:42:12.901065 8827 tablet.cc:2369] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:42:12.940053 8693 tablet.cc:2369] T ca08d1cac1d3490c8aa7e0ec15d0966a P 60aedfc7473b494aa0fe8224a34ddc6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:42:12.971266 8559 tablet.cc:2369] T ca08d1cac1d3490c8aa7e0ec15d0966a P 734f68f73bb74817b3eb1124a43fc30e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:42:13.021696 8849 raft_consensus.cc:482] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:13.021941 8849 raft_consensus.cc:504] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37379 } } peers { permanent_uuid: "60aedfc7473b494aa0fe8224a34ddc6b" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 44553 } } peers { permanent_uuid: "734f68f73bb74817b3eb1124a43fc30e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 46629 } }
I20260929 18:42:13.023293 8849 leader_election.cc:288] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60aedfc7473b494aa0fe8224a34ddc6b (127.7.157.66:44553), 734f68f73bb74817b3eb1124a43fc30e (127.7.157.65:46629)
I20260929 18:42:13.028184 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:13.023675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:46655 (local address 127.7.157.65:46629)
0929 18:42:13.023808 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:13.023812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:13.024074 (+ 262us) server_negotiation.cc:407] Connection header received
0929 18:42:13.024168 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:13.024171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:13.024221 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:13.024277 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:13.024500 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:13.025233 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:13.026836 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:13.027488 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:13.027626 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:13.027683 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:13.027973 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 18:42:13.028049 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:42:13.028184 8842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:13.023591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:47243 (local address 127.7.157.66:44553)
0929 18:42:13.023711 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:13.023717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:13.023793 (+ 76us) server_negotiation.cc:407] Connection header received
0929 18:42:13.023949 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:13.023952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:13.024000 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:13.024080 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:13.024348 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:13.025157 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:13.026836 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:13.027515 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:13.027625 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:13.027683 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:13.027973 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 18:42:13.028048 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:42:13.028218 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:13.023499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:44553 (local address 127.7.157.67:47243)
0929 18:42:13.023708 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:13.023723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:13.023829 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:13.024114 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:13.024117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:13.024130 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:13.024243 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:13.024248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:13.025273 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:13.025277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:13.026682 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:13.026694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:13.027637 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:13.027641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:13.027826 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:13.027865 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 18:42:13.027949 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:42:13.028049 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":118}
I20260929 18:42:13.028225 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:13.023583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:46629 (local address 127.7.157.67:46655)
0929 18:42:13.023985 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:13.023996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:13.024075 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:13.024304 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:13.024306 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:13.024322 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:13.024408 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:13.024412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:13.025308 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:13.025311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:13.026682 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:13.026694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:13.027637 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:13.027641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:13.027804 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:13.027865 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 18:42:13.027949 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:42:13.028048 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":115,"thread_start_us":114,"threads_started":1}
I20260929 18:42:13.028923 8512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca08d1cac1d3490c8aa7e0ec15d0966a" candidate_uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734f68f73bb74817b3eb1124a43fc30e" is_pre_election: true
I20260929 18:42:13.028918 8646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca08d1cac1d3490c8aa7e0ec15d0966a" candidate_uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60aedfc7473b494aa0fe8224a34ddc6b" is_pre_election: true
I20260929 18:42:13.029207 8646 raft_consensus.cc:2454] T ca08d1cac1d3490c8aa7e0ec15d0966a P 60aedfc7473b494aa0fe8224a34ddc6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5ddf2e9c39d43a09ba7fe8b778598ff in term 0.
I20260929 18:42:13.029207 8512 raft_consensus.cc:2454] T ca08d1cac1d3490c8aa7e0ec15d0966a P 734f68f73bb74817b3eb1124a43fc30e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5ddf2e9c39d43a09ba7fe8b778598ff in term 0.
I20260929 18:42:13.029665 8716 leader_election.cc:302] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60aedfc7473b494aa0fe8224a34ddc6b, d5ddf2e9c39d43a09ba7fe8b778598ff; no voters:
I20260929 18:42:13.029924 8849 raft_consensus.cc:2781] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:13.030021 8849 raft_consensus.cc:482] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:13.030064 8849 raft_consensus.cc:3037] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:13.031087 8849 raft_consensus.cc:504] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37379 } } peers { permanent_uuid: "60aedfc7473b494aa0fe8224a34ddc6b" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 44553 } } peers { permanent_uuid: "734f68f73bb74817b3eb1124a43fc30e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 46629 } }
I20260929 18:42:13.031448 8849 leader_election.cc:288] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [CANDIDATE]: Term 1 election: Requested vote from peers 60aedfc7473b494aa0fe8224a34ddc6b (127.7.157.66:44553), 734f68f73bb74817b3eb1124a43fc30e (127.7.157.65:46629)
I20260929 18:42:13.031817 8512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca08d1cac1d3490c8aa7e0ec15d0966a" candidate_uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734f68f73bb74817b3eb1124a43fc30e"
I20260929 18:42:13.031826 8646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca08d1cac1d3490c8aa7e0ec15d0966a" candidate_uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60aedfc7473b494aa0fe8224a34ddc6b"
I20260929 18:42:13.031944 8646 raft_consensus.cc:3037] T ca08d1cac1d3490c8aa7e0ec15d0966a P 60aedfc7473b494aa0fe8224a34ddc6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:13.031939 8512 raft_consensus.cc:3037] T ca08d1cac1d3490c8aa7e0ec15d0966a P 734f68f73bb74817b3eb1124a43fc30e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:13.032933 8512 raft_consensus.cc:2454] T ca08d1cac1d3490c8aa7e0ec15d0966a P 734f68f73bb74817b3eb1124a43fc30e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ddf2e9c39d43a09ba7fe8b778598ff in term 1.
I20260929 18:42:13.032964 8646 raft_consensus.cc:2454] T ca08d1cac1d3490c8aa7e0ec15d0966a P 60aedfc7473b494aa0fe8224a34ddc6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ddf2e9c39d43a09ba7fe8b778598ff in term 1.
I20260929 18:42:13.033252 8714 leader_election.cc:302] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 734f68f73bb74817b3eb1124a43fc30e, d5ddf2e9c39d43a09ba7fe8b778598ff; no voters:
I20260929 18:42:13.033417 8849 raft_consensus.cc:2781] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:13.033625 8849 raft_consensus.cc:686] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [term 1 LEADER]: Becoming Leader. State: Replica: d5ddf2e9c39d43a09ba7fe8b778598ff, State: Running, Role: LEADER
I20260929 18:42:13.033974 8849 consensus_queue.cc:237] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37379 } } peers { permanent_uuid: "60aedfc7473b494aa0fe8224a34ddc6b" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 44553 } } peers { permanent_uuid: "734f68f73bb74817b3eb1124a43fc30e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 46629 } }
I20260929 18:42:13.037449 8366 catalog_manager.cc:5495] T ca08d1cac1d3490c8aa7e0ec15d0966a P d5ddf2e9c39d43a09ba7fe8b778598ff reported cstate change: term changed from 0 to 1, leader changed from <none> to d5ddf2e9c39d43a09ba7fe8b778598ff (127.7.157.67). New cstate: current_term: 1 leader_uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ddf2e9c39d43a09ba7fe8b778598ff" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60aedfc7473b494aa0fe8224a34ddc6b" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 44553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "734f68f73bb74817b3eb1124a43fc30e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 46629 } health_report { overall_health: UNKNOWN } } }
Sep 29 18:42:13 dist-test-slave-smpn krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.65@KRBTEST.COM
I20260929 18:42:13.074154 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:13.065619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33968 (local address 127.7.157.65:46629)
0929 18:42:13.065774 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:13.065779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:13.065919 (+ 140us) server_negotiation.cc:407] Connection header received
0929 18:42:13.066057 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:13.066061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:13.066107 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:13.066176 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:13.066875 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:13.067432 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:13.068277 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:13.068493 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:13.068583 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:13.070943 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:13.070957 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:13.070973 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:13.070997 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:13.072436 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:13.072849 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:13.072855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:13.072856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:13.072941 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:13.073178 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:13.073181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:13.073183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:13.073439 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:13.073640 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:13.073872 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 18:42:13.073991 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 18:42:13 dist-test-slave-smpn krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707333, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:42:13 dist-test-slave-smpn krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707333, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.157.65@KRBTEST.COM
I20260929 18:42:13.099562 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:13.091493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33972 (local address 127.7.157.65:46629)
0929 18:42:13.091643 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:13.091648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:13.091796 (+ 148us) server_negotiation.cc:407] Connection header received
0929 18:42:13.091935 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:13.091938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:13.091984 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:13.092062 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:13.092803 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:13.093336 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:13.094150 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:13.094375 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:13.094463 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:13.096780 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:13.096783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:13.096785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:13.096813 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:13.097928 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:13.098311 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:13.098314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:13.098315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:13.098392 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:13.098641 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:13.098644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:13.098646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:13.098821 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:13.098937 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:13.099281 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 18:42:13.099404 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 18:42:13 dist-test-slave-smpn krb5kdc[8322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:42:13 dist-test-slave-smpn krb5kdc[8322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.65@KRBTEST.COM
I20260929 18:42:13.123953 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:13.116040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.7.157.65:46629)
0929 18:42:13.116183 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:13.116188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:13.116371 (+ 183us) server_negotiation.cc:407] Connection header received
0929 18:42:13.116522 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:13.116525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:13.116584 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:13.116649 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:13.117356 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:13.117862 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:13.118696 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:13.118904 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:13.118992 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:13.121413 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:13.121417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:13.121419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:13.121448 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:13.122572 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:13.122930 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:13.122932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:13.122934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:13.122999 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:13.123248 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:13.123250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:13.123251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:13.123413 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:13.123506 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:13.123692 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 18:42:13.123791 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 18:42:13.125365 8472 tablet_service.cc:2908] Rejecting scan request for tablet ca08d1cac1d3490c8aa7e0ec15d0966a: Uninitialized: clean time has not yet been initialized
I20260929 18:42:13.127395 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 8429
I20260929 18:42:13.132583 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 8563
I20260929 18:42:13.137418 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 8697
I20260929 18:42:13.142216 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 8335
2026-09-29T18:42:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2455 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:42:13 dist-test-slave-smpn krb5kdc[8867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:42:13 dist-test-slave-smpn krb5kdc[8867](info): set up 2 sockets
Sep 29 18:42:13 dist-test-slave-smpn krb5kdc[8867](info): commencing operation
krb5kdc: starting...
W20260929 18:42:14.114573 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:42:14 dist-test-slave-smpn krb5kdc[8867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:42:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:42:14.229492 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:44357
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:39723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:44357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:14.340408 8880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:14.340667 8880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:14.340708 8880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:14.344769 8880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:14.344826 8880 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:42:14.344842 8880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:14.344858 8880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:14.344875 8880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:14.344905 8880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:14.350273 8880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:39723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:44357
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:44357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:14.351432 8880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:14.352488 8880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:14.361016 8888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:14.361064 8885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:14.361025 8886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:14.361079 8880 server_base.cc:1034] running on GCE node
I20260929 18:42:14.361871 8880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:14.362968 8880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:14.364121 8880 hybrid_clock.cc:648] HybridClock initialized: now 1790707334364106 us; error 30 us; skew 500 ppm
Sep 29 18:42:14 dist-test-slave-smpn krb5kdc[8867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707334, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:14.366772 8880 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:42:14.367726 8880 webserver.cc:460] Webserver started at http://127.7.157.126:33731/ using document root <none> and password file <none>
I20260929 18:42:14.368259 8880 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:14.368319 8880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:14.368505 8880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:14.370533 8880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "75ed5efb2af04ea8adf11b43ad9465c7"
format_stamp: "Formatted at 2026-09-29 18:42:14 on dist-test-slave-smpn"
server_key: "fe239d364dc72d9b2e441f935470c38f"
server_key_iv: "c833ede0aa651162ce9ac8f00be4dc71"
server_key_version: "encryptionkey@0"
I20260929 18:42:14.371104 8880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "75ed5efb2af04ea8adf11b43ad9465c7"
format_stamp: "Formatted at 2026-09-29 18:42:14 on dist-test-slave-smpn"
server_key: "fe239d364dc72d9b2e441f935470c38f"
server_key_iv: "c833ede0aa651162ce9ac8f00be4dc71"
server_key_version: "encryptionkey@0"
I20260929 18:42:14.374812 8880 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:42:14.377223 8895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:14.378445 8880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:42:14.378579 8880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "75ed5efb2af04ea8adf11b43ad9465c7"
format_stamp: "Formatted at 2026-09-29 18:42:14 on dist-test-slave-smpn"
server_key: "fe239d364dc72d9b2e441f935470c38f"
server_key_iv: "c833ede0aa651162ce9ac8f00be4dc71"
server_key_version: "encryptionkey@0"
I20260929 18:42:14.378679 8880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:14.400275 8880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:14.403008 8880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:14.403205 8880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:14.410692 8880 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:44357
I20260929 18:42:14.410707 8957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:44357 every 8 connection(s)
I20260929 18:42:14.411659 8880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:14.414444 8958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:14.415661 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 8880
I20260929 18:42:14.415781 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:14.416054 7797 external_mini_cluster.cc:1428] Setting key d409b71c67ed07b1046e35b97e5ae9a5
I20260929 18:42:14.420789 8958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7: Bootstrap starting.
I20260929 18:42:14.423193 8958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:14.423920 8958 log.cc:824] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:42:14 dist-test-slave-smpn krb5kdc[8867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:14.425961 8958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7: No bootstrap required, opened a new log
I20260929 18:42:14.429013 8958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ed5efb2af04ea8adf11b43ad9465c7" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44357 } }
I20260929 18:42:14.429296 8958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:14.429404 8958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ed5efb2af04ea8adf11b43ad9465c7, State: Initialized, Role: FOLLOWER
I20260929 18:42:14.429857 8958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ed5efb2af04ea8adf11b43ad9465c7" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44357 } }
I20260929 18:42:14.430002 8958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:14.430068 8958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:14.430166 8958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:14.430191 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:14.417516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50018 (local address 127.7.157.126:44357)
0929 18:42:14.417961 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:14.417981 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:14.418025 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:42:14.418676 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:14.418695 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:14.419033 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:14.419386 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:14.420160 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.421250 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.422078 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.422550 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.422691 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:14.425236 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:14.425241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:14.425255 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:14.425280 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:14.427339 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:14.427795 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:14.427800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:14.427806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:14.427923 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:14.428189 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:14.428191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:14.428192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:14.428596 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:14.428783 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:14.429041 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 18:42:14.429351 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":138,"threads_started":1}
I20260929 18:42:14.431346 8958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ed5efb2af04ea8adf11b43ad9465c7" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44357 } }
I20260929 18:42:14.431676 8958 leader_election.cc:302] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75ed5efb2af04ea8adf11b43ad9465c7; no voters:
I20260929 18:42:14.431977 8958 leader_election.cc:288] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:14.432090 8963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:14.432585 8963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [term 1 LEADER]: Becoming Leader. State: Replica: 75ed5efb2af04ea8adf11b43ad9465c7, State: Running, Role: LEADER
I20260929 18:42:14.432919 8963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ed5efb2af04ea8adf11b43ad9465c7" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44357 } }
I20260929 18:42:14.433285 8958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:14.434568 8965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75ed5efb2af04ea8adf11b43ad9465c7. Latest consensus state: current_term: 1 leader_uuid: "75ed5efb2af04ea8adf11b43ad9465c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ed5efb2af04ea8adf11b43ad9465c7" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44357 } } }
I20260929 18:42:14.434708 8965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:14.435022 8964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75ed5efb2af04ea8adf11b43ad9465c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ed5efb2af04ea8adf11b43ad9465c7" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44357 } } }
I20260929 18:42:14.435134 8964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:14.435158 8974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:14.438238 8974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:14.443825 8974 catalog_manager.cc:1347] Generated new cluster ID: 5c425d31d5e94c7881c337fb63d5f758
I20260929 18:42:14.443899 8974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:14.450380 8974 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:14.451797 8974 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:14.469340 8974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7: Generated new TSK 0
I20260929 18:42:14.470224 8974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:42:14.496309 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:44357
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:39723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:14.606864 8984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:14.607182 8984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:14.607231 8984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:14.611193 8984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:14.611264 8984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:14.611322 8984 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:42:14.611374 8984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:42:14.616675 8984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:39723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.157.126:44357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:14.617713 8984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:14.618796 8984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:14.628328 8992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:14.628466 8984 server_base.cc:1034] running on GCE node
W20260929 18:42:14.628316 8989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:14.628299 8990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:14.628957 8984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:14.629598 8984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:14.630757 8984 hybrid_clock.cc:648] HybridClock initialized: now 1790707334630738 us; error 31 us; skew 500 ppm
Sep 29 18:42:14 dist-test-slave-smpn krb5kdc[8867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707334, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:14.633390 8984 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:42:14.634315 8984 webserver.cc:460] Webserver started at http://127.7.157.65:43457/ using document root <none> and password file <none>
I20260929 18:42:14.634861 8984 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:14.634928 8984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:14.635167 8984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:14.637161 8984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "1901f021a3304ba0a03d14de0746731f"
format_stamp: "Formatted at 2026-09-29 18:42:14 on dist-test-slave-smpn"
server_key: "b377e01373d6ac386d6bcd1af5bc87fd"
server_key_iv: "f049a2c412c2ac2dd2e4e73599881b51"
server_key_version: "encryptionkey@0"
I20260929 18:42:14.637670 8984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "1901f021a3304ba0a03d14de0746731f"
format_stamp: "Formatted at 2026-09-29 18:42:14 on dist-test-slave-smpn"
server_key: "b377e01373d6ac386d6bcd1af5bc87fd"
server_key_iv: "f049a2c412c2ac2dd2e4e73599881b51"
server_key_version: "encryptionkey@0"
I20260929 18:42:14.641260 8984 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:42:14.643610 8999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:14.644786 8984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:14.644884 8984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "1901f021a3304ba0a03d14de0746731f"
format_stamp: "Formatted at 2026-09-29 18:42:14 on dist-test-slave-smpn"
server_key: "b377e01373d6ac386d6bcd1af5bc87fd"
server_key_iv: "f049a2c412c2ac2dd2e4e73599881b51"
server_key_version: "encryptionkey@0"
I20260929 18:42:14.644992 8984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:14.662348 8984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:14.665419 8984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:14.665588 8984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:14.667128 8984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:14.667178 8984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:14.667222 8984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:14.667251 8984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:42:14 dist-test-slave-smpn krb5kdc[8867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707334, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:14.677382 8984 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:34413
I20260929 18:42:14.677428 9114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:34413 every 8 connection(s)
I20260929 18:42:14.678462 8984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:42:14.681234 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:14.668094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:50885 (local address 127.7.157.126:44357)
0929 18:42:14.668239 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:14.668244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:14.669247 (+ 1003us) server_negotiation.cc:407] Connection header received
0929 18:42:14.670499 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:14.670503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:14.670576 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:14.670671 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:14.671950 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.672578 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.674036 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.674405 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.674520 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:14.677726 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:14.677731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:14.677734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:14.677765 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:14.679279 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:14.679779 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:14.679783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:14.679785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:14.679867 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:14.680181 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:14.680184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:14.680186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:14.680367 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:14.680454 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:14.680934 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 18:42:14.681072 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:42:14.682044 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:14.668576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:44357 (local address 127.7.157.65:50885)
0929 18:42:14.669040 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:14.669136 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:14.670239 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:14.670832 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:14.670840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:14.671294 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:14.671795 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:14.671809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.672707 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.672710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:14.673914 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:14.673926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.674713 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.674718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:14.674934 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:14.675678 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:14.675701 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:14.677505 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:14.679387 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:14.679393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:14.679409 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:14.679683 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:14.679949 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:14.679952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:14.679954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:14.680087 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:14.680465 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:14.680472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:14.680713 (+ 241us) client_negotiation.cc:769] Sending connection context
0929 18:42:14.680876 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 18:42:14.681198 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260929 18:42:14.682597 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 8984
I20260929 18:42:14.682709 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:42:14.683022 7797 external_mini_cluster.cc:1428] Setting key 995dca3959fc86124741e730df96add7
I20260929 18:42:14.683455 9115 heartbeater.cc:347] Connected to a master server at 127.7.157.126:44357
I20260929 18:42:14.683629 9115 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:14.684185 9115 heartbeater.cc:511] Master 127.7.157.126:44357 requested a full tablet report, sending...
I20260929 18:42:14.685662 8911 ts_manager.cc:194] Registered new tserver with Master: 1901f021a3304ba0a03d14de0746731f (127.7.157.65:34413)
I20260929 18:42:14.687189 8911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:50885
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
I20260929 18:42:14.692407 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:14.686253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:32989 (local address 127.7.157.126:44357)
0929 18:42:14.686364 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:14.686367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:14.686377 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:42:14.686404 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:14.686406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:14.686438 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:14.686502 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:14.686922 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.687532 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.688419 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.688644 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.688725 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:14.689732 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:14.689736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:14.689739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:14.689762 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:14.690985 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:14.691384 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:14.691387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:14.691389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:14.691459 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:14.691692 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:14.691695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:14.691696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:14.691850 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:14.691931 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:14.692169 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 18:42:14.692262 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:42:14.692423 9121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:14.685499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:44357 (local address 127.7.157.65:32989)
0929 18:42:14.685782 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:14.685796 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:14.685878 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:14.686552 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:14.686555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:14.686729 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:14.686819 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:14.686825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.687642 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.687646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:14.688314 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:14.688323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.688737 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.688741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:14.688826 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:14.689281 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:14.689295 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:14.689612 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:14.691117 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:14.691120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:14.691122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:14.691301 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:14.691536 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:14.691539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:14.691541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:14.691619 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:14.691932 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:14.691935 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:14.692020 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 18:42:14.692110 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:42:14.692242 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:42:14.708006 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:44357
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:39723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:14.819051 9124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:14.819386 9124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:14.819453 9124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:14.823607 9124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:14.823688 9124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:14.823745 9124 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:42:14.823801 9124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:42:14.829347 9124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:39723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.157.126:44357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:14.830521 9124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:14.831655 9124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:14.841686 9132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:14.841686 9129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:14.841881 9124 server_base.cc:1034] running on GCE node
W20260929 18:42:14.841684 9130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:14.842339 9124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:14.843015 9124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:14.844202 9124 hybrid_clock.cc:648] HybridClock initialized: now 1790707334844178 us; error 34 us; skew 500 ppm
Sep 29 18:42:14 dist-test-slave-smpn krb5kdc[8867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707334, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:14.850687 9124 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:42:14.851645 9124 webserver.cc:460] Webserver started at http://127.7.157.66:39317/ using document root <none> and password file <none>
I20260929 18:42:14.852183 9124 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:14.852248 9124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:14.852443 9124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:14.854513 9124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "07e15950a18845789e6fb0d5a0c544c7"
format_stamp: "Formatted at 2026-09-29 18:42:14 on dist-test-slave-smpn"
server_key: "b160b7ce37182bdbc18c9dbfbfd8953f"
server_key_iv: "c98194ed8e225205c20839ea0573ec47"
server_key_version: "encryptionkey@0"
I20260929 18:42:14.855111 9124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "07e15950a18845789e6fb0d5a0c544c7"
format_stamp: "Formatted at 2026-09-29 18:42:14 on dist-test-slave-smpn"
server_key: "b160b7ce37182bdbc18c9dbfbfd8953f"
server_key_iv: "c98194ed8e225205c20839ea0573ec47"
server_key_version: "encryptionkey@0"
I20260929 18:42:14.858850 9124 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:42:14.861265 9139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:14.862529 9124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:14.862650 9124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "07e15950a18845789e6fb0d5a0c544c7"
format_stamp: "Formatted at 2026-09-29 18:42:14 on dist-test-slave-smpn"
server_key: "b160b7ce37182bdbc18c9dbfbfd8953f"
server_key_iv: "c98194ed8e225205c20839ea0573ec47"
server_key_version: "encryptionkey@0"
I20260929 18:42:14.862746 9124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:14.889449 9124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:14.892333 9124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:14.892511 9124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:14.893945 9124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:14.893994 9124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:14.894047 9124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:14.894075 9124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:14.904259 9124 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:37005
Sep 29 18:42:14 dist-test-slave-smpn krb5kdc[8867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707334, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:14.904330 9254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:37005 every 8 connection(s)
I20260929 18:42:14.905159 9124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:42:14.908329 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:14.894809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:33965 (local address 127.7.157.126:44357)
0929 18:42:14.894919 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:14.894922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:14.896059 (+ 1137us) server_negotiation.cc:407] Connection header received
0929 18:42:14.897101 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:14.897106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:14.897165 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:14.897248 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:14.898930 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.899649 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.901414 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.901643 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.901729 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:14.904950 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:14.904953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:14.904956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:14.904983 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:14.906473 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:14.906931 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:14.906934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:14.906935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:14.907009 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:14.907332 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:14.907334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:14.907336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:14.907513 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:14.907608 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:14.908105 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 18:42:14.908206 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:42:14.909221 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:14.895430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:44357 (local address 127.7.157.66:33965)
0929 18:42:14.895911 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:14.895956 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:14.896874 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:14.897469 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:14.897480 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:14.897969 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:14.898726 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:14.898749 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.899767 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.899771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:14.901298 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:14.901309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.901805 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.901810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:14.902020 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:14.902772 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:14.902795 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:14.904757 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:14.906589 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:14.906593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:14.906605 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:14.906851 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:14.907122 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:14.907125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:14.907127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:14.907261 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:14.907635 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:14.907641 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:14.907898 (+ 257us) client_negotiation.cc:769] Sending connection context
0929 18:42:14.908065 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 18:42:14.908334 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":135,"threads_started":1}
I20260929 18:42:14.910368 9255 heartbeater.cc:347] Connected to a master server at 127.7.157.126:44357
I20260929 18:42:14.910526 9255 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:14.911024 9255 heartbeater.cc:511] Master 127.7.157.126:44357 requested a full tablet report, sending...
I20260929 18:42:14.912175 8912 ts_manager.cc:194] Registered new tserver with Master: 07e15950a18845789e6fb0d5a0c544c7 (127.7.157.66:37005)
I20260929 18:42:14.912870 8912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:33965
I20260929 18:42:14.913985 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 9124
I20260929 18:42:14.914100 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:42:14.914381 7797 external_mini_cluster.cc:1428] Setting key 9b4a9de41d3201f1eba6b79595f2bf15
I20260929 18:42:14.919623 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:14.912455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:40955 (local address 127.7.157.126:44357)
0929 18:42:14.912568 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:14.912572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:14.912745 (+ 173us) server_negotiation.cc:407] Connection header received
0929 18:42:14.912843 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:14.912845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:14.912898 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:14.912969 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:14.913412 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.913994 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.914968 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.915225 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.915305 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:14.916413 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:14.916417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:14.916420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:14.916441 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:14.917674 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:14.918134 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:14.918139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:14.918140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:14.918242 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:14.918549 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:14.918552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:14.918572 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:14.918964 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:14.919088 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:14.919353 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 18:42:14.919472 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:42:14.919703 9261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:14.912376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:44357 (local address 127.7.157.66:40955)
0929 18:42:14.912678 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:14.912689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:14.912765 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:14.913000 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:14.913003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:14.913202 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:14.913314 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:14.913320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.914149 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.914153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:14.914876 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:14.914885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:14.915330 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:14.915333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:14.915420 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:14.915870 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:14.915883 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:14.916288 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:14.917776 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:14.917779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:14.917781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:14.918020 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:14.918342 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:14.918346 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:14.918348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:14.918434 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:14.919055 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:14.919084 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:14.919180 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 18:42:14.919284 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 18:42:14.919427 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:42:14.941496 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:44357
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:39723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:15.052171 9264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:15.052453 9264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:15.052515 9264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:15.056589 9264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:15.056659 9264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:15.056711 9264 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:42:15.056766 9264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:42:15.062183 9264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:39723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.157.126:44357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:15.063401 9264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:15.064481 9264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:15.074182 9272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:15.074167 9269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:15.074126 9270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:15.074147 9264 server_base.cc:1034] running on GCE node
I20260929 18:42:15.074752 9264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:15.075472 9264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:15.076627 9264 hybrid_clock.cc:648] HybridClock initialized: now 1790707335076603 us; error 37 us; skew 500 ppm
Sep 29 18:42:15 dist-test-slave-smpn krb5kdc[8867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707335, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:15.079365 9264 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:42:15.080222 9264 webserver.cc:460] Webserver started at http://127.7.157.67:35595/ using document root <none> and password file <none>
I20260929 18:42:15.080762 9264 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:15.080819 9264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:15.080988 9264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:15.083200 9264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "a8d9ed0bc7964809a4f25f32dd8acad2"
format_stamp: "Formatted at 2026-09-29 18:42:15 on dist-test-slave-smpn"
server_key: "17a1d2c87de9a607ed9d8c2c25ac9f08"
server_key_iv: "035bd0df520d9ef4382e62df6f548c3c"
server_key_version: "encryptionkey@0"
I20260929 18:42:15.083768 9264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "a8d9ed0bc7964809a4f25f32dd8acad2"
format_stamp: "Formatted at 2026-09-29 18:42:15 on dist-test-slave-smpn"
server_key: "17a1d2c87de9a607ed9d8c2c25ac9f08"
server_key_iv: "035bd0df520d9ef4382e62df6f548c3c"
server_key_version: "encryptionkey@0"
I20260929 18:42:15.087635 9264 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:42:15.090040 9279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:15.091405 9264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:15.091514 9264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "a8d9ed0bc7964809a4f25f32dd8acad2"
format_stamp: "Formatted at 2026-09-29 18:42:15 on dist-test-slave-smpn"
server_key: "17a1d2c87de9a607ed9d8c2c25ac9f08"
server_key_iv: "035bd0df520d9ef4382e62df6f548c3c"
server_key_version: "encryptionkey@0"
I20260929 18:42:15.091627 9264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:15.110363 9264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:15.113756 9264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:15.113938 9264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:15.115527 9264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:15.115578 9264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:15.115653 9264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:15.115682 9264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:42:15 dist-test-slave-smpn krb5kdc[8867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707335, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:15.126245 9264 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:46309
I20260929 18:42:15.126310 9394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:46309 every 8 connection(s)
I20260929 18:42:15.127435 9264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:42:15.129516 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.116281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:51635 (local address 127.7.157.126:44357)
0929 18:42:15.116440 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:15.116446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:15.117378 (+ 932us) server_negotiation.cc:407] Connection header received
0929 18:42:15.119343 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:15.119348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:15.119450 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:15.119549 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:15.120921 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.121453 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.122686 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.122908 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.123006 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.125850 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:15.125854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:15.125857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:15.125887 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:15.127544 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:15.128139 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:15.128143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:15.128145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:15.128241 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:15.128586 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:15.128588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:15.128589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:15.128770 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:15.128846 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:15.129287 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 18:42:15.129394 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:42:15.130338 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.116694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:44357 (local address 127.7.157.67:51635)
0929 18:42:15.117199 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:15.117263 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:15.118589 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:15.119755 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:15.119762 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:15.120230 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:15.120770 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.120783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.121574 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.121577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.122568 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.122578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.123095 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.123100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.123265 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.123863 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:15.123879 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:15.125657 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:15.127683 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:15.127688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:15.127701 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:15.127997 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:15.128324 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:15.128327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:15.128329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:15.128500 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:15.128845 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:15.128850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:15.129083 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 18:42:15.129252 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 18:42:15.129507 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":106,"threads_started":1}
I20260929 18:42:15.131546 9395 heartbeater.cc:347] Connected to a master server at 127.7.157.126:44357
I20260929 18:42:15.131696 9395 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:15.132195 9395 heartbeater.cc:511] Master 127.7.157.126:44357 requested a full tablet report, sending...
I20260929 18:42:15.133204 8911 ts_manager.cc:194] Registered new tserver with Master: a8d9ed0bc7964809a4f25f32dd8acad2 (127.7.157.67:46309)
I20260929 18:42:15.133885 8911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:51635
I20260929 18:42:15.137640 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 9264
I20260929 18:42:15.137759 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:42:15.138038 7797 external_mini_cluster.cc:1428] Setting key 3d8bf8e257c38c2dc7b7a6060f86b522
I20260929 18:42:15.140134 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:42:15.140203 9401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.133442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:44357 (local address 127.7.157.67:46933)
0929 18:42:15.133726 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:15.133738 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:15.133836 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:15.134078 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:15.134081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:15.134260 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:15.134351 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.134357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.135148 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.135151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.135939 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.135949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.136313 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.136315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.136389 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.136842 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:15.136853 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:15.137154 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:15.138734 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:15.138737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:15.138738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:15.138958 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:15.139266 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:15.139268 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:15.139270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:15.139359 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:15.139701 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:15.139704 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:15.139799 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 18:42:15.139894 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 18:42:15.140037 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":76,"threads_started":1}
I20260929 18:42:15.140221 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.133492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:46933 (local address 127.7.157.126:44357)
0929 18:42:15.133621 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:15.133645 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:15.133812 (+ 167us) server_negotiation.cc:407] Connection header received
0929 18:42:15.133908 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:15.133910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:15.133956 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:15.134023 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:15.134457 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.135023 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.136030 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.136235 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.136306 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.137248 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:15.137250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:15.137252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:15.137267 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:15.138644 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:15.139041 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:15.139044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:15.139046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:15.139172 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:15.139446 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:15.139449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:15.139450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:15.139619 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:15.139693 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:15.139960 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 18:42:15.140071 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 18:42:15 dist-test-slave-smpn krb5kdc[8867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:42:15 dist-test-slave-smpn krb5kdc[8867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:15.164016 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.156548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50030 (local address 127.7.157.126:44357)
0929 18:42:15.156681 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:15.156685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:15.156696 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:42:15.156736 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:15.156738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:15.156774 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:15.156840 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:15.157517 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.158024 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.158858 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.159100 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.159190 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.161327 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:15.161330 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:15.161332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:15.161355 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:15.162482 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:15.162863 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:15.162867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:15.162869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:15.162942 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:15.163218 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:15.163221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:15.163223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:15.163398 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:15.163515 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:15.163751 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 18:42:15.163864 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:42:15.166805 8911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:42:15.169730 8911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:15.184199 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.178226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.7.157.67:46309)
0929 18:42:15.178875 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:15.178885 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:15.178904 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:42:15.178966 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:15.178970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:15.179216 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:15.179379 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:15.179956 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.181131 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.182678 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.183520 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.183640 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.183811 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:15.183950 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 18:42:15.184048 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":121,"threads_started":1}
I20260929 18:42:15.184248 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.179159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:34413 (local address 127.0.0.1:50410)
0929 18:42:15.179379 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:15.179391 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:15.179461 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:15.180196 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:15.180199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:15.180215 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:15.180305 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.180311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.181501 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.181505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.182889 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.182897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.183709 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.183712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.183784 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.183810 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:42:15.183896 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:42:15.184041 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":167,"thread_start_us":94,"threads_started":1}
I20260929 18:42:15.184252 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.178838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:37005 (local address 127.0.0.1:41486)
0929 18:42:15.179052 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:15.179090 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:15.179171 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:15.180475 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:15.180478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:15.180493 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:15.180594 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.180601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.181625 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.181628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.182768 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.182779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.183861 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.183865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.183964 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.183986 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:42:15.184032 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 18:42:15.184105 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":4,"thread_start_us":80,"threads_started":1}
I20260929 18:42:15.184461 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.179345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50410 (local address 127.7.157.65:34413)
0929 18:42:15.179735 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:15.179751 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:15.179774 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:42:15.179843 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:15.179848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:15.180064 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:15.180218 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:15.180410 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.181402 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.182980 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.183578 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.184016 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.184195 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:15.184245 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 18:42:15.184341 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":88,"threads_started":1}
I20260929 18:42:15.184753 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.179660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41486 (local address 127.7.157.66:37005)
0929 18:42:15.180076 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:15.180087 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:15.180106 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:42:15.180170 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:15.180175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:15.180368 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:15.180503 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:15.180688 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.181533 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.182908 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.183718 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.184342 (+ 624us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.184493 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:15.184537 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 18:42:15.184617 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":107,"threads_started":1}
I20260929 18:42:15.184845 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.178116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:46309 (local address 127.0.0.1:50128)
0929 18:42:15.178495 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:15.178552 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:15.178708 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:15.179633 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:15.179639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:15.179663 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:15.179802 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.179809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.181241 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.181244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.182542 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.182552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.183652 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.183656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.183766 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.183810 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 18:42:15.183916 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 18:42:15.184733 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260929 18:42:15.187250 9329 tablet_service.cc:1467] Processing CreateTablet for tablet eb1ed06dd61041ce96e041e9d73cd7f1 (DEFAULT_TABLE table=test-table [id=23f853fcc7c24d219178ff9396f3959a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:15.187249 9049 tablet_service.cc:1467] Processing CreateTablet for tablet eb1ed06dd61041ce96e041e9d73cd7f1 (DEFAULT_TABLE table=test-table [id=23f853fcc7c24d219178ff9396f3959a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:15.188179 9329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1ed06dd61041ce96e041e9d73cd7f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:15.188189 9049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1ed06dd61041ce96e041e9d73cd7f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:15.189738 9189 tablet_service.cc:1467] Processing CreateTablet for tablet eb1ed06dd61041ce96e041e9d73cd7f1 (DEFAULT_TABLE table=test-table [id=23f853fcc7c24d219178ff9396f3959a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:15.190748 9189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1ed06dd61041ce96e041e9d73cd7f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:15.195168 9418 tablet_bootstrap.cc:492] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2: Bootstrap starting.
I20260929 18:42:15.195892 9419 tablet_bootstrap.cc:492] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f: Bootstrap starting.
I20260929 18:42:15.197341 9418 tablet_bootstrap.cc:654] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:15.198077 9419 tablet_bootstrap.cc:654] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:15.198076 9418 log.cc:824] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:15.199292 9419 log.cc:824] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:15.200245 9418 tablet_bootstrap.cc:492] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2: No bootstrap required, opened a new log
I20260929 18:42:15.200448 9418 ts_tablet_manager.cc:1397] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:42:15.201166 9419 tablet_bootstrap.cc:492] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f: No bootstrap required, opened a new log
I20260929 18:42:15.201314 9419 ts_tablet_manager.cc:1397] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 18:42:15.203313 9421 tablet_bootstrap.cc:492] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7: Bootstrap starting.
I20260929 18:42:15.203523 9418 raft_consensus.cc:348] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } }
I20260929 18:42:15.203759 9418 raft_consensus.cc:374] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:15.203827 9418 raft_consensus.cc:729] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d9ed0bc7964809a4f25f32dd8acad2, State: Initialized, Role: FOLLOWER
I20260929 18:42:15.204272 9418 consensus_queue.cc:260] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } }
I20260929 18:42:15.204409 9419 raft_consensus.cc:348] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } }
I20260929 18:42:15.204692 9419 raft_consensus.cc:374] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:15.204747 9419 raft_consensus.cc:729] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1901f021a3304ba0a03d14de0746731f, State: Initialized, Role: FOLLOWER
I20260929 18:42:15.205227 9418 ts_tablet_manager.cc:1428] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:42:15.205169 9419 consensus_queue.cc:260] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } }
I20260929 18:42:15.205581 9395 heartbeater.cc:503] Master 127.7.157.126:44357 was elected leader, sending a full tablet report...
I20260929 18:42:15.205799 9421 tablet_bootstrap.cc:654] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:15.206105 9419 ts_tablet_manager.cc:1428] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260929 18:42:15.206506 9115 heartbeater.cc:503] Master 127.7.157.126:44357 was elected leader, sending a full tablet report...
I20260929 18:42:15.206666 9421 log.cc:824] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:15.209197 9421 tablet_bootstrap.cc:492] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7: No bootstrap required, opened a new log
I20260929 18:42:15.209393 9421 ts_tablet_manager.cc:1397] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260929 18:42:15.212983 9421 raft_consensus.cc:348] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } }
I20260929 18:42:15.213253 9421 raft_consensus.cc:374] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:15.213321 9421 raft_consensus.cc:729] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e15950a18845789e6fb0d5a0c544c7, State: Initialized, Role: FOLLOWER
I20260929 18:42:15.213757 9421 consensus_queue.cc:260] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } }
I20260929 18:42:15.214408 9255 heartbeater.cc:503] Master 127.7.157.126:44357 was elected leader, sending a full tablet report...
I20260929 18:42:15.214756 9421 ts_tablet_manager.cc:1428] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 18:42:15.378692 9396 tablet.cc:2369] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:42:15.406535 9256 tablet.cc:2369] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:42:15.426709 9426 raft_consensus.cc:482] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:15.426893 9426 raft_consensus.cc:504] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } }
I20260929 18:42:15.428002 9426 leader_election.cc:288] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8d9ed0bc7964809a4f25f32dd8acad2 (127.7.157.67:46309), 1901f021a3304ba0a03d14de0746731f (127.7.157.65:34413)
W20260929 18:42:15.429772 9116 tablet.cc:2369] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:42:15.432065 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.428712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:51741 (local address 127.7.157.65:34413)
0929 18:42:15.428836 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:15.428840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:15.428919 (+ 79us) server_negotiation.cc:407] Connection header received
0929 18:42:15.429011 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:15.429013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:15.429047 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:15.429116 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:15.429325 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.430017 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.431124 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.431521 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.431632 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.431686 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:15.431878 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 18:42:15.431945 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:42:15.432078 9427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.428235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:46309 (local address 127.7.157.66:57671)
0929 18:42:15.428469 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:15.428484 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:15.428615 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:15.428890 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:15.428894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:15.428914 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:15.429041 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.429047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.429999 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.430002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.430974 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.430983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.431632 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.431636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.431774 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.431797 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:42:15.431870 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:42:15.431940 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":76,"thread_start_us":71,"threads_started":1}
I20260929 18:42:15.432093 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.428382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:57671 (local address 127.7.157.67:46309)
0929 18:42:15.428508 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:15.428513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:15.428560 (+ 47us) server_negotiation.cc:407] Connection header received
0929 18:42:15.428728 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:15.428732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:15.428782 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:15.428865 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:15.429140 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.429905 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.431108 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.431520 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.431640 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.431690 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:15.431892 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 18:42:15.431961 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:42:15.432086 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.428575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:34413 (local address 127.7.157.66:51741)
0929 18:42:15.428844 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:15.428856 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:15.428930 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:15.429142 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:15.429145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:15.429162 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:15.429240 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.429244 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.430119 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.430123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.430983 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.430994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.431630 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.431634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.431764 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.431793 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 18:42:15.431861 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:42:15.431928 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":29,"thread_start_us":69,"threads_started":1}
I20260929 18:42:15.432763 9349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1ed06dd61041ce96e041e9d73cd7f1" candidate_uuid: "07e15950a18845789e6fb0d5a0c544c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" is_pre_election: true
I20260929 18:42:15.433009 9349 raft_consensus.cc:2454] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07e15950a18845789e6fb0d5a0c544c7 in term 0.
I20260929 18:42:15.433471 9143 leader_election.cc:302] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07e15950a18845789e6fb0d5a0c544c7, a8d9ed0bc7964809a4f25f32dd8acad2; no voters:
I20260929 18:42:15.433696 9069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1ed06dd61041ce96e041e9d73cd7f1" candidate_uuid: "07e15950a18845789e6fb0d5a0c544c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1901f021a3304ba0a03d14de0746731f" is_pre_election: true
I20260929 18:42:15.433717 9426 raft_consensus.cc:2781] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:15.433787 9426 raft_consensus.cc:482] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:15.433817 9426 raft_consensus.cc:3037] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:15.433939 9069 raft_consensus.cc:2454] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07e15950a18845789e6fb0d5a0c544c7 in term 0.
I20260929 18:42:15.434947 9426 raft_consensus.cc:504] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } }
I20260929 18:42:15.435293 9426 leader_election.cc:288] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [CANDIDATE]: Term 1 election: Requested vote from peers a8d9ed0bc7964809a4f25f32dd8acad2 (127.7.157.67:46309), 1901f021a3304ba0a03d14de0746731f (127.7.157.65:34413)
I20260929 18:42:15.435559 9349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1ed06dd61041ce96e041e9d73cd7f1" candidate_uuid: "07e15950a18845789e6fb0d5a0c544c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2"
I20260929 18:42:15.435600 9069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1ed06dd61041ce96e041e9d73cd7f1" candidate_uuid: "07e15950a18845789e6fb0d5a0c544c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1901f021a3304ba0a03d14de0746731f"
I20260929 18:42:15.435657 9349 raft_consensus.cc:3037] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:15.435732 9069 raft_consensus.cc:3037] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:15.436678 9069 raft_consensus.cc:2454] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07e15950a18845789e6fb0d5a0c544c7 in term 1.
I20260929 18:42:15.436923 9349 raft_consensus.cc:2454] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07e15950a18845789e6fb0d5a0c544c7 in term 1.
I20260929 18:42:15.436945 9143 leader_election.cc:302] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07e15950a18845789e6fb0d5a0c544c7, 1901f021a3304ba0a03d14de0746731f; no voters:
I20260929 18:42:15.437193 9426 raft_consensus.cc:2781] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:15.437364 9426 raft_consensus.cc:686] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [term 1 LEADER]: Becoming Leader. State: Replica: 07e15950a18845789e6fb0d5a0c544c7, State: Running, Role: LEADER
I20260929 18:42:15.437655 9426 consensus_queue.cc:237] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } }
I20260929 18:42:15.440541 8911 catalog_manager.cc:5495] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07e15950a18845789e6fb0d5a0c544c7 (127.7.157.66). New cstate: current_term: 1 leader_uuid: "07e15950a18845789e6fb0d5a0c544c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } health_report { overall_health: UNKNOWN } } }
I20260929 18:42:15.480160 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.476512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.7.157.66:37005)
0929 18:42:15.476642 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:15.476647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:15.476662 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:42:15.476874 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:15.476878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:15.476935 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:15.477018 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:15.477283 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.477792 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.478676 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.478897 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.478984 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.479349 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:15.479778 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:15.479879 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:15.479961 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 18:42:15.480021 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:42:15.485971 9349 raft_consensus.cc:1261] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 1 FOLLOWER]: Refusing update from remote peer 07e15950a18845789e6fb0d5a0c544c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:15.485972 9069 raft_consensus.cc:1261] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f [term 1 FOLLOWER]: Refusing update from remote peer 07e15950a18845789e6fb0d5a0c544c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:15.486603 9429 consensus_queue.cc:1035] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:15.486783 9426 consensus_queue.cc:1035] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:15.495247 9433 mvcc.cc:205] Tried to move back new op lower bound from 7334737246143963136 to 7334737245955059712. Current Snapshot: MvccSnapshot[applied={T|T < 7334737246143963136}]
I20260929 18:42:15.505770 8912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:15.505939 8912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:15.508322 8912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7: Sending DeleteTablet for 3 replicas of tablet eb1ed06dd61041ce96e041e9d73cd7f1
I20260929 18:42:15.509120 9329 tablet_service.cc:1514] Processing DeleteTablet for tablet eb1ed06dd61041ce96e041e9d73cd7f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:15 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:50128
I20260929 18:42:15.509233 9049 tablet_service.cc:1514] Processing DeleteTablet for tablet eb1ed06dd61041ce96e041e9d73cd7f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:15 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:50410
I20260929 18:42:15.509285 9189 tablet_service.cc:1514] Processing DeleteTablet for tablet eb1ed06dd61041ce96e041e9d73cd7f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:15 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:41486
I20260929 18:42:15.509711 9444 tablet_replica.cc:330] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f: stopping tablet replica
I20260929 18:42:15.510015 9444 raft_consensus.cc:2229] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:15.510322 9444 raft_consensus.cc:2258] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:15.510399 9443 tablet_replica.cc:330] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7: stopping tablet replica
I20260929 18:42:15.510730 9443 raft_consensus.cc:2229] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:42:15.510736 8912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:42:15.510903 9442 tablet_replica.cc:330] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2: stopping tablet replica
I20260929 18:42:15.511124 9443 raft_consensus.cc:2258] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 18:42:15.511226 8912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:15.511250 9442 raft_consensus.cc:2229] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:15.511538 9442 raft_consensus.cc:2258] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:15.512809 9444 ts_tablet_manager.cc:1905] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:15.513051 9443 ts_tablet_manager.cc:1905] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:15.515754 9443 ts_tablet_manager.cc:1918] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:15.515890 9443 log.cc:1196] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/eb1ed06dd61041ce96e041e9d73cd7f1
I20260929 18:42:15.515988 9444 ts_tablet_manager.cc:1918] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:15.516144 9444 log.cc:1196] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/wal/wals/eb1ed06dd61041ce96e041e9d73cd7f1
I20260929 18:42:15.516239 9443 ts_tablet_manager.cc:1939] T eb1ed06dd61041ce96e041e9d73cd7f1 P 07e15950a18845789e6fb0d5a0c544c7: Deleting consensus metadata
I20260929 18:42:15.516505 9444 ts_tablet_manager.cc:1939] T eb1ed06dd61041ce96e041e9d73cd7f1 P 1901f021a3304ba0a03d14de0746731f: Deleting consensus metadata
I20260929 18:42:15.516947 9049 tablet_service.cc:1467] Processing CreateTablet for tablet ce8108b6d1da4e09923a241517bb2c86 (DEFAULT_TABLE table=test-table [id=4d667a25faf14427813c5f5613ac5740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:15.517194 9049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce8108b6d1da4e09923a241517bb2c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:15.517670 9189 tablet_service.cc:1467] Processing CreateTablet for tablet ce8108b6d1da4e09923a241517bb2c86 (DEFAULT_TABLE table=test-table [id=4d667a25faf14427813c5f5613ac5740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:15.517879 9189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce8108b6d1da4e09923a241517bb2c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:15.519014 9442 ts_tablet_manager.cc:1905] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:15.519305 9419 tablet_bootstrap.cc:492] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f: Bootstrap starting.
I20260929 18:42:15.520437 9329 tablet_service.cc:1467] Processing CreateTablet for tablet ce8108b6d1da4e09923a241517bb2c86 (DEFAULT_TABLE table=test-table [id=4d667a25faf14427813c5f5613ac5740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:15.520694 9329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce8108b6d1da4e09923a241517bb2c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:15.521370 9421 tablet_bootstrap.cc:492] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7: Bootstrap starting.
I20260929 18:42:15.521541 9442 ts_tablet_manager.cc:1918] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:15.521633 9442 log.cc:1196] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/eb1ed06dd61041ce96e041e9d73cd7f1
I20260929 18:42:15.521911 9442 ts_tablet_manager.cc:1939] T eb1ed06dd61041ce96e041e9d73cd7f1 P a8d9ed0bc7964809a4f25f32dd8acad2: Deleting consensus metadata
I20260929 18:42:15.522352 9421 tablet_bootstrap.cc:654] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:15.522851 9418 tablet_bootstrap.cc:492] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2: Bootstrap starting.
I20260929 18:42:15.523258 8899 catalog_manager.cc:4841] TS 1901f021a3304ba0a03d14de0746731f (127.7.157.65:34413): tablet eb1ed06dd61041ce96e041e9d73cd7f1 (table test-table [id=23f853fcc7c24d219178ff9396f3959a]) successfully deleted
I20260929 18:42:15.523517 8899 catalog_manager.cc:4841] TS 07e15950a18845789e6fb0d5a0c544c7 (127.7.157.66:37005): tablet eb1ed06dd61041ce96e041e9d73cd7f1 (table test-table [id=23f853fcc7c24d219178ff9396f3959a]) successfully deleted
I20260929 18:42:15.523581 9419 tablet_bootstrap.cc:654] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:15.523689 8899 catalog_manager.cc:4841] TS a8d9ed0bc7964809a4f25f32dd8acad2 (127.7.157.67:46309): tablet eb1ed06dd61041ce96e041e9d73cd7f1 (table test-table [id=23f853fcc7c24d219178ff9396f3959a]) successfully deleted
I20260929 18:42:15.523757 9418 tablet_bootstrap.cc:654] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:15.524863 9418 tablet_bootstrap.cc:492] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2: No bootstrap required, opened a new log
I20260929 18:42:15.524900 9421 tablet_bootstrap.cc:492] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7: No bootstrap required, opened a new log
I20260929 18:42:15.524942 9418 ts_tablet_manager.cc:1397] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:15.524972 9421 ts_tablet_manager.cc:1397] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 18:42:15.525259 9419 tablet_bootstrap.cc:492] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f: No bootstrap required, opened a new log
I20260929 18:42:15.525332 9419 ts_tablet_manager.cc:1397] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260929 18:42:15.525373 9418 raft_consensus.cc:348] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:15.525400 9421 raft_consensus.cc:348] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:15.525472 9418 raft_consensus.cc:374] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:15.525501 9421 raft_consensus.cc:374] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:15.525504 9418 raft_consensus.cc:729] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d9ed0bc7964809a4f25f32dd8acad2, State: Initialized, Role: FOLLOWER
I20260929 18:42:15.525529 9421 raft_consensus.cc:729] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e15950a18845789e6fb0d5a0c544c7, State: Initialized, Role: FOLLOWER
I20260929 18:42:15.525650 9421 consensus_queue.cc:260] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:15.525643 9418 consensus_queue.cc:260] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:15.525884 9421 ts_tablet_manager.cc:1428] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:15.525884 9418 ts_tablet_manager.cc:1428] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:15.526176 9419 raft_consensus.cc:348] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:15.526288 9419 raft_consensus.cc:374] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:15.526319 9419 raft_consensus.cc:729] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1901f021a3304ba0a03d14de0746731f, State: Initialized, Role: FOLLOWER
I20260929 18:42:15.526438 9419 consensus_queue.cc:260] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:15.526690 9419 ts_tablet_manager.cc:1428] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:42:15.785633 9423 raft_consensus.cc:482] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:15.785778 9423 raft_consensus.cc:504] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:15.786770 9423 leader_election.cc:288] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1901f021a3304ba0a03d14de0746731f (127.7.157.65:34413), 07e15950a18845789e6fb0d5a0c544c7 (127.7.157.66:37005)
I20260929 18:42:15.791769 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.787494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:47605 (local address 127.7.157.66:37005)
0929 18:42:15.787607 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:15.787611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:15.787749 (+ 138us) server_negotiation.cc:407] Connection header received
0929 18:42:15.787844 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:15.787846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:15.787887 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:15.787943 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:15.788173 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.788856 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.790482 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.791125 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.791240 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.791292 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:15.791567 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 18:42:15.791636 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 18:42:15.791769 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.787125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:53053 (local address 127.7.157.65:34413)
0929 18:42:15.787236 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:15.787240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:15.787358 (+ 118us) server_negotiation.cc:407] Connection header received
0929 18:42:15.787606 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:15.787611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:15.787654 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:15.787714 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:15.787991 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.788753 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.790482 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.791118 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.791240 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.791292 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:15.791567 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 18:42:15.791649 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:42:15.791812 9448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.787009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:34413 (local address 127.7.157.67:53053)
0929 18:42:15.787280 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:15.787293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:15.787406 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:15.787747 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:15.787751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:15.787764 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:15.787886 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.787892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.788851 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.788853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.790342 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.790354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.791232 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.791236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.791430 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.791478 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 18:42:15.791548 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:42:15.791641 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":174,"thread_start_us":105,"threads_started":1}
I20260929 18:42:15.791821 9449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.787398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:37005 (local address 127.7.157.67:47605)
0929 18:42:15.787644 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:15.787656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:15.787761 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:15.787969 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:15.787972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:15.787989 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:15.788081 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.788085 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.788938 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.788940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.790342 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.790354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.791232 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.791236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.791436 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.791474 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 18:42:15.791548 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 18:42:15.791641 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":145,"thread_start_us":69,"threads_started":1}
I20260929 18:42:15.792493 9069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce8108b6d1da4e09923a241517bb2c86" candidate_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1901f021a3304ba0a03d14de0746731f" is_pre_election: true
I20260929 18:42:15.792505 9209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce8108b6d1da4e09923a241517bb2c86" candidate_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07e15950a18845789e6fb0d5a0c544c7" is_pre_election: true
I20260929 18:42:15.792629 9069 raft_consensus.cc:2454] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8d9ed0bc7964809a4f25f32dd8acad2 in term 0.
I20260929 18:42:15.792737 9209 raft_consensus.cc:2454] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8d9ed0bc7964809a4f25f32dd8acad2 in term 0.
I20260929 18:42:15.793037 9283 leader_election.cc:302] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1901f021a3304ba0a03d14de0746731f, a8d9ed0bc7964809a4f25f32dd8acad2; no voters:
I20260929 18:42:15.793304 9423 raft_consensus.cc:2781] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:15.793361 9423 raft_consensus.cc:482] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:15.793387 9423 raft_consensus.cc:3037] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:15.794579 9423 raft_consensus.cc:504] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:15.794931 9423 leader_election.cc:288] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [CANDIDATE]: Term 1 election: Requested vote from peers 1901f021a3304ba0a03d14de0746731f (127.7.157.65:34413), 07e15950a18845789e6fb0d5a0c544c7 (127.7.157.66:37005)
I20260929 18:42:15.795310 9069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce8108b6d1da4e09923a241517bb2c86" candidate_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1901f021a3304ba0a03d14de0746731f"
I20260929 18:42:15.795325 9209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce8108b6d1da4e09923a241517bb2c86" candidate_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07e15950a18845789e6fb0d5a0c544c7"
I20260929 18:42:15.795439 9209 raft_consensus.cc:3037] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:15.795426 9069 raft_consensus.cc:3037] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:15.796399 9209 raft_consensus.cc:2454] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8d9ed0bc7964809a4f25f32dd8acad2 in term 1.
I20260929 18:42:15.796407 9069 raft_consensus.cc:2454] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8d9ed0bc7964809a4f25f32dd8acad2 in term 1.
I20260929 18:42:15.796643 9283 leader_election.cc:302] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1901f021a3304ba0a03d14de0746731f, a8d9ed0bc7964809a4f25f32dd8acad2; no voters:
I20260929 18:42:15.796811 9423 raft_consensus.cc:2781] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:15.796983 9423 raft_consensus.cc:686] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 1 LEADER]: Becoming Leader. State: Replica: a8d9ed0bc7964809a4f25f32dd8acad2, State: Running, Role: LEADER
I20260929 18:42:15.797250 9423 consensus_queue.cc:237] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:15.799526 8910 catalog_manager.cc:5495] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8d9ed0bc7964809a4f25f32dd8acad2 (127.7.157.67). New cstate: current_term: 1 leader_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } health_report { overall_health: HEALTHY } } }
Sep 29 18:42:15 dist-test-slave-smpn krb5kdc[8867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707334, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:15.856787 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.848850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:44357 (local address 127.0.0.1:59992)
0929 18:42:15.849193 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:15.849210 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:15.849348 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:15.849628 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:15.849630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:15.850019 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:15.850131 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.850137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.850838 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.850840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.851451 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:15.851460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.851825 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.851827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:15.851908 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.852405 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:15.852417 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:15.853773 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:15.855363 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:15.855372 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:15.855378 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:15.855652 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:15.855883 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:15.855886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:15.855887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:15.855970 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:15.856287 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:15.856293 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:15.856397 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 18:42:15.856484 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:42:15.856622 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":16,"thread_start_us":143,"threads_started":1}
I20260929 18:42:15.856804 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:15.848885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.7.157.126:44357)
0929 18:42:15.849222 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:15.849229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:15.849305 (+ 76us) server_negotiation.cc:407] Connection header received
0929 18:42:15.849452 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:15.849454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:15.849495 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:15.849567 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:15.850229 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.850751 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.851531 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:15.851745 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:15.851813 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:15.853898 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:15.853902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:15.853904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:15.853924 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:15.855269 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:15.855727 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:15.855729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:15.855731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:15.855798 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:15.856036 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:15.856038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:15.856039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:15.856214 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:15.856281 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:15.856529 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 18:42:15.856674 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":282,"thread_start_us":114,"threads_started":1}
I20260929 18:42:15.860462 8910 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:59992:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:42:15.861200 8910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:15.865724 9189 tablet_service.cc:1467] Processing CreateTablet for tablet 0a9122fd4354411ba546f5c42011015c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1657b1324d6745829a7faa414f300015]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:42:15.865727 9049 tablet_service.cc:1467] Processing CreateTablet for tablet 0a9122fd4354411ba546f5c42011015c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1657b1324d6745829a7faa414f300015]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:42:15.865727 9329 tablet_service.cc:1467] Processing CreateTablet for tablet 0a9122fd4354411ba546f5c42011015c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1657b1324d6745829a7faa414f300015]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:42:15.866109 9329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9122fd4354411ba546f5c42011015c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:15.866109 9049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9122fd4354411ba546f5c42011015c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:15.866110 9189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9122fd4354411ba546f5c42011015c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:15.868595 9421 tablet_bootstrap.cc:492] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7: Bootstrap starting.
I20260929 18:42:15.869011 9418 tablet_bootstrap.cc:492] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2: Bootstrap starting.
I20260929 18:42:15.869182 9419 tablet_bootstrap.cc:492] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f: Bootstrap starting.
I20260929 18:42:15.869470 9421 tablet_bootstrap.cc:654] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:15.870396 9418 tablet_bootstrap.cc:654] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:15.870525 9419 tablet_bootstrap.cc:654] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:15.871145 9421 tablet_bootstrap.cc:492] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7: No bootstrap required, opened a new log
I20260929 18:42:15.871237 9421 ts_tablet_manager.cc:1397] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:42:15.871645 9418 tablet_bootstrap.cc:492] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2: No bootstrap required, opened a new log
I20260929 18:42:15.871724 9418 ts_tablet_manager.cc:1397] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:42:15.871670 9421 raft_consensus.cc:348] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:15.871781 9421 raft_consensus.cc:374] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:15.871814 9421 raft_consensus.cc:729] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e15950a18845789e6fb0d5a0c544c7, State: Initialized, Role: FOLLOWER
I20260929 18:42:15.871896 9419 tablet_bootstrap.cc:492] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f: No bootstrap required, opened a new log
I20260929 18:42:15.871968 9419 ts_tablet_manager.cc:1397] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 18:42:15.872136 9418 raft_consensus.cc:348] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:15.872145 9421 consensus_queue.cc:260] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:15.872239 9418 raft_consensus.cc:374] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:15.872260 9418 raft_consensus.cc:729] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d9ed0bc7964809a4f25f32dd8acad2, State: Initialized, Role: FOLLOWER
I20260929 18:42:15.872390 9419 raft_consensus.cc:348] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:15.872493 9419 raft_consensus.cc:374] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:15.872426 9439 tablet_replica.cc:439] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } }
I20260929 18:42:15.872524 9419 raft_consensus.cc:729] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1901f021a3304ba0a03d14de0746731f, State: Initialized, Role: FOLLOWER
I20260929 18:42:15.872556 9439 tablet_replica.cc:442] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:15.872573 9418 consensus_queue.cc:260] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:15.872701 9419 consensus_queue.cc:260] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:15.872778 9423 tablet_replica.cc:439] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } }
I20260929 18:42:15.872885 9423 tablet_replica.cc:442] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:15.872984 9419 ts_tablet_manager.cc:1428] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260929 18:42:15.873214 9424 tablet_replica.cc:439] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } }
I20260929 18:42:15.873329 9424 tablet_replica.cc:442] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:15.873632 9421 ts_tablet_manager.cc:1428] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:15.873935 9418 ts_tablet_manager.cc:1428] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:16.021076 9423 raft_consensus.cc:482] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:16.021203 9423 raft_consensus.cc:504] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:16.021559 9423 leader_election.cc:288] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1901f021a3304ba0a03d14de0746731f (127.7.157.65:34413), 07e15950a18845789e6fb0d5a0c544c7 (127.7.157.66:37005)
I20260929 18:42:16.021970 9069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9122fd4354411ba546f5c42011015c" candidate_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1901f021a3304ba0a03d14de0746731f" is_pre_election: true
I20260929 18:42:16.022101 9069 raft_consensus.cc:2454] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8d9ed0bc7964809a4f25f32dd8acad2 in term 0.
I20260929 18:42:16.022046 9209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9122fd4354411ba546f5c42011015c" candidate_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07e15950a18845789e6fb0d5a0c544c7" is_pre_election: true
I20260929 18:42:16.022219 9209 raft_consensus.cc:2454] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8d9ed0bc7964809a4f25f32dd8acad2 in term 0.
I20260929 18:42:16.022434 9283 leader_election.cc:302] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1901f021a3304ba0a03d14de0746731f, a8d9ed0bc7964809a4f25f32dd8acad2; no voters:
I20260929 18:42:16.022583 9423 raft_consensus.cc:2781] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:16.022643 9423 raft_consensus.cc:482] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:16.022670 9423 raft_consensus.cc:3037] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:16.023687 9423 raft_consensus.cc:504] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:16.023991 9423 leader_election.cc:288] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [CANDIDATE]: Term 1 election: Requested vote from peers 1901f021a3304ba0a03d14de0746731f (127.7.157.65:34413), 07e15950a18845789e6fb0d5a0c544c7 (127.7.157.66:37005)
I20260929 18:42:16.024330 9069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9122fd4354411ba546f5c42011015c" candidate_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1901f021a3304ba0a03d14de0746731f"
I20260929 18:42:16.024374 9209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9122fd4354411ba546f5c42011015c" candidate_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07e15950a18845789e6fb0d5a0c544c7"
I20260929 18:42:16.024436 9069 raft_consensus.cc:3037] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:16.024495 9209 raft_consensus.cc:3037] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:16.025367 9069 raft_consensus.cc:2454] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8d9ed0bc7964809a4f25f32dd8acad2 in term 1.
I20260929 18:42:16.025374 9209 raft_consensus.cc:2454] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8d9ed0bc7964809a4f25f32dd8acad2 in term 1.
I20260929 18:42:16.025624 9283 leader_election.cc:302] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07e15950a18845789e6fb0d5a0c544c7, a8d9ed0bc7964809a4f25f32dd8acad2; no voters:
I20260929 18:42:16.025784 9423 raft_consensus.cc:2781] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:16.025861 9423 raft_consensus.cc:686] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [term 1 LEADER]: Becoming Leader. State: Replica: a8d9ed0bc7964809a4f25f32dd8acad2, State: Running, Role: LEADER
I20260929 18:42:16.025993 9423 consensus_queue.cc:237] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } }
I20260929 18:42:16.026607 9450 tablet_replica.cc:439] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2: TxnStatusTablet state changed. Reason: New leader a8d9ed0bc7964809a4f25f32dd8acad2. Latest consensus state: current_term: 1 leader_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } }
I20260929 18:42:16.026695 9450 tablet_replica.cc:442] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:42:16.026887 9463 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:42:16.027047 9463 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:42:16.027693 8910 catalog_manager.cc:5495] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8d9ed0bc7964809a4f25f32dd8acad2 (127.7.157.67). New cstate: current_term: 1 leader_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } health_report { overall_health: HEALTHY } } }
I20260929 18:42:16.066380 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:16.061861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.7.157.67:46309)
0929 18:42:16.062187 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:16.062192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:16.062425 (+ 233us) server_negotiation.cc:407] Connection header received
0929 18:42:16.062630 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:16.062632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:16.062686 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:16.062782 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:16.063042 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:16.063620 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:16.064843 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:16.065075 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:16.065150 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:16.065477 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:16.065943 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:16.066026 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:16.066136 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 18:42:16.066197 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":95,"threads_started":1}
I20260929 18:42:16.066376 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:16.062122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:46309 (local address 127.0.0.1:59186)
0929 18:42:16.062341 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:16.062356 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:16.062485 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:16.062814 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:16.062817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:16.062825 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:42:16.062939 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:16.062945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:16.063754 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:16.063758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:16.064728 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:16.064740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:16.065175 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:16.065180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:16.065273 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:16.065335 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:42:16.066023 (+ 688us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:42:16.066039 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 18:42:16.066114 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:42:16.066230 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 18:42:16.069736 9069 raft_consensus.cc:1261] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f [term 1 FOLLOWER]: Refusing update from remote peer a8d9ed0bc7964809a4f25f32dd8acad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:16.069875 9209 raft_consensus.cc:1261] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7 [term 1 FOLLOWER]: Refusing update from remote peer a8d9ed0bc7964809a4f25f32dd8acad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:16.070241 9423 consensus_queue.cc:1035] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:16.070436 9450 consensus_queue.cc:1035] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:16.071970 9424 tablet_replica.cc:439] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f: TxnStatusTablet state changed. Reason: New leader a8d9ed0bc7964809a4f25f32dd8acad2. Latest consensus state: current_term: 1 leader_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } }
I20260929 18:42:16.072098 9424 tablet_replica.cc:442] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:16.072206 9439 tablet_replica.cc:439] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7: TxnStatusTablet state changed. Reason: New leader a8d9ed0bc7964809a4f25f32dd8acad2. Latest consensus state: current_term: 1 leader_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } }
I20260929 18:42:16.072316 9439 tablet_replica.cc:442] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:16.074697 9423 tablet_replica.cc:439] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } }
I20260929 18:42:16.074808 9423 tablet_replica.cc:442] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:42:16.075034 9439 tablet_replica.cc:439] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } }
I20260929 18:42:16.075166 9439 tablet_replica.cc:442] T 0a9122fd4354411ba546f5c42011015c P 07e15950a18845789e6fb0d5a0c544c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:16.077010 9469 mvcc.cc:205] Tried to move back new op lower bound from 7334737248536244224 to 7334737248363700224. Current Snapshot: MvccSnapshot[applied={T|T < 7334737248536244224}]
I20260929 18:42:16.082720 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:16.079520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.7.157.67:46309)
0929 18:42:16.079631 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:16.079634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:16.079840 (+ 206us) server_negotiation.cc:407] Connection header received
0929 18:42:16.079956 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:16.079959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:16.080007 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:16.080066 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:16.080287 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:16.080789 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:16.081660 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:16.081944 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:16.082028 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:16.082277 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:16.082348 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:16.082402 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:16.082524 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 18:42:16.082572 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:42:16.084570 9423 tablet_replica.cc:439] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } }
I20260929 18:42:16.084697 9423 tablet_replica.cc:442] T 0a9122fd4354411ba546f5c42011015c P a8d9ed0bc7964809a4f25f32dd8acad2: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:42:16.085476 9424 tablet_replica.cc:439] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 } } peers { permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 } } peers { permanent_uuid: "a8d9ed0bc7964809a4f25f32dd8acad2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46309 } } }
I20260929 18:42:16.085593 9424 tablet_replica.cc:442] T 0a9122fd4354411ba546f5c42011015c P 1901f021a3304ba0a03d14de0746731f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:16.086050 9468 mvcc.cc:205] Tried to move back new op lower bound from 7334737248536244224 to 7334737248363700224. Current Snapshot: MvccSnapshot[applied={T|T < 7334737248536244224}]
I20260929 18:42:16.095723 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:16.091428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:37499 (local address 127.7.157.67:46309)
0929 18:42:16.091536 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:16.091539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:16.091758 (+ 219us) server_negotiation.cc:407] Connection header received
0929 18:42:16.091880 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:16.091882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:16.091919 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:16.091977 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:16.092263 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:16.092759 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:16.093983 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:16.094197 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:16.094272 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:16.095049 (+ 777us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:16.095298 (+ 249us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:16.095390 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:16.095506 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 18:42:16.095556 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":39}
I20260929 18:42:16.095762 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:16.091361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:46309 (local address 127.7.157.67:37499)
0929 18:42:16.091679 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:16.091693 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:16.091807 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:16.092006 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:16.092009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:16.092017 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:42:16.092177 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:16.092183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:16.092858 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:16.092862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:16.093869 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:16.093879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:16.094380 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:16.094384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:16.094662 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:16.094768 (+ 106us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:42:16.095410 (+ 642us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:42:16.095417 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:42:16.095494 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:42:16.095575 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":17,"thread_start_us":84,"threads_started":1}
I20260929 18:42:16.103016 9209 raft_consensus.cc:1261] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7 [term 1 FOLLOWER]: Refusing update from remote peer a8d9ed0bc7964809a4f25f32dd8acad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:16.103147 9069 raft_consensus.cc:1261] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f [term 1 FOLLOWER]: Refusing update from remote peer a8d9ed0bc7964809a4f25f32dd8acad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:16.103430 9450 consensus_queue.cc:1035] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07e15950a18845789e6fb0d5a0c544c7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:16.103660 9423 consensus_queue.cc:1035] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1901f021a3304ba0a03d14de0746731f" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:16.149623 8910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:16.149799 8910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:16.152097 8910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75ed5efb2af04ea8adf11b43ad9465c7: Sending DeleteTablet for 3 replicas of tablet ce8108b6d1da4e09923a241517bb2c86
I20260929 18:42:16.152768 9189 tablet_service.cc:1514] Processing DeleteTablet for tablet ce8108b6d1da4e09923a241517bb2c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:16 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:41486
I20260929 18:42:16.152907 9049 tablet_service.cc:1514] Processing DeleteTablet for tablet ce8108b6d1da4e09923a241517bb2c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:16 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:50410
I20260929 18:42:16.153432 9328 tablet_service.cc:1514] Processing DeleteTablet for tablet ce8108b6d1da4e09923a241517bb2c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:16 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:50128
I20260929 18:42:16.153769 9486 tablet_replica.cc:330] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2: stopping tablet replica
I20260929 18:42:16.153893 9486 raft_consensus.cc:2229] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:42:16.154171 9486 raft_consensus.cc:2258] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:16.154680 9485 tablet_replica.cc:330] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f: stopping tablet replica
I20260929 18:42:16.154814 9486 ts_tablet_manager.cc:1905] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:16.154954 9485 raft_consensus.cc:2229] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:16.155160 9485 raft_consensus.cc:2258] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:16.155490 9484 tablet_replica.cc:330] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7: stopping tablet replica
I20260929 18:42:16.156108 9485 ts_tablet_manager.cc:1905] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:16.157442 9486 ts_tablet_manager.cc:1918] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:42:16.157509 9486 log.cc:1196] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/ce8108b6d1da4e09923a241517bb2c86
I20260929 18:42:16.157764 9486 ts_tablet_manager.cc:1939] T ce8108b6d1da4e09923a241517bb2c86 P a8d9ed0bc7964809a4f25f32dd8acad2: Deleting consensus metadata
I20260929 18:42:16.158387 8899 catalog_manager.cc:4841] TS a8d9ed0bc7964809a4f25f32dd8acad2 (127.7.157.67:46309): tablet ce8108b6d1da4e09923a241517bb2c86 (table test-table [id=4d667a25faf14427813c5f5613ac5740]) successfully deleted
I20260929 18:42:16.159389 9485 ts_tablet_manager.cc:1918] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:42:16.159544 9484 raft_consensus.cc:2229] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:16.159749 9484 raft_consensus.cc:2258] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:16.159957 9485 log.cc:1196] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-0/wal/wals/ce8108b6d1da4e09923a241517bb2c86
I20260929 18:42:16.160486 9484 ts_tablet_manager.cc:1905] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:16.162050 9485 ts_tablet_manager.cc:1939] T ce8108b6d1da4e09923a241517bb2c86 P 1901f021a3304ba0a03d14de0746731f: Deleting consensus metadata
I20260929 18:42:16.162786 8899 catalog_manager.cc:4841] TS 1901f021a3304ba0a03d14de0746731f (127.7.157.65:34413): tablet ce8108b6d1da4e09923a241517bb2c86 (table test-table [id=4d667a25faf14427813c5f5613ac5740]) successfully deleted
I20260929 18:42:16.164631 9484 ts_tablet_manager.cc:1918] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:42:16.164722 9484 log.cc:1196] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/ce8108b6d1da4e09923a241517bb2c86
Sep 29 18:42:16 dist-test-slave-smpn krb5kdc[8867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.157.65@KRBTEST.COM
I20260929 18:42:16.165124 9484 ts_tablet_manager.cc:1939] T ce8108b6d1da4e09923a241517bb2c86 P 07e15950a18845789e6fb0d5a0c544c7: Deleting consensus metadata
I20260929 18:42:16.166167 8899 catalog_manager.cc:4841] TS 07e15950a18845789e6fb0d5a0c544c7 (127.7.157.66:37005): tablet ce8108b6d1da4e09923a241517bb2c86 (table test-table [id=4d667a25faf14427813c5f5613ac5740]) successfully deleted
I20260929 18:42:16.169371 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:16.154767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.7.157.65:34413)
0929 18:42:16.154886 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:16.154890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:16.159345 (+ 4455us) server_negotiation.cc:407] Connection header received
0929 18:42:16.159457 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:16.159460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:16.159509 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:16.159586 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:16.160567 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:16.161076 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:16.161864 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:16.162074 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:16.162175 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:16.165421 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:16.165432 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:16.165445 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:16.165471 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:16.167516 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:16.167960 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:16.167968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:16.167970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:16.168074 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:16.168393 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:16.168397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:16.168398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:16.168744 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:16.168975 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:16.169077 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 18:42:16.169204 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 18:42:16.169870 9109 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56244
I20260929 18:42:16.180078 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:16.171851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60002 (local address 127.7.157.126:44357)
0929 18:42:16.172032 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:16.172036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:16.172329 (+ 293us) server_negotiation.cc:407] Connection header received
0929 18:42:16.172450 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:16.172454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:16.172513 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:16.172604 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:16.173252 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:16.174107 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:16.174947 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:16.175316 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:16.175412 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:16.176803 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:16.176807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:16.176810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:16.176835 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:16.178412 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:16.178821 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:16.178824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:16.178826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:16.178910 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:16.179226 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:16.179229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:16.179231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:16.179446 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:16.179570 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:16.179764 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 18:42:16.179902 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 18:42:16.180502 8910 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60002
I20260929 18:42:16.181843 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 8984
I20260929 18:42:16.189162 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 9124
I20260929 18:42:16.197034 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 9264
I20260929 18:42:16.202955 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 8880
2026-09-29T18:42:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3061 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:42:16 dist-test-slave-smpn krb5kdc[9493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:42:16 dist-test-slave-smpn krb5kdc[9493](info): set up 2 sockets
Sep 29 18:42:16 dist-test-slave-smpn krb5kdc[9493](info): commencing operation
krb5kdc: starting...
W20260929 18:42:17.162673 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:42:17 dist-test-slave-smpn krb5kdc[9493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:42:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:42:17.276000 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:37709
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:37709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:17.384094 9506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:17.384367 9506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:17.384416 9506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:17.388381 9506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:17.388443 9506 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:42:17.388459 9506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:17.388476 9506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:17.388499 9506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:17.388518 9506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:17.393755 9506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:37709
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:37709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:17.394783 9506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:17.395800 9506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:42:17.404229 9506 server_base.cc:1034] running on GCE node
W20260929 18:42:17.404229 9514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:17.404356 9511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:17.404431 9512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:17.405045 9506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:17.406263 9506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:17.407420 9506 hybrid_clock.cc:648] HybridClock initialized: now 1790707337407384 us; error 53 us; skew 500 ppm
Sep 29 18:42:17 dist-test-slave-smpn krb5kdc[9493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707337, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:17.410038 9506 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:42:17.410889 9506 webserver.cc:460] Webserver started at http://127.7.157.126:35253/ using document root <none> and password file <none>
I20260929 18:42:17.411470 9506 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:17.411540 9506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:17.411746 9506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:17.413712 9506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "0016b60f610a4161b1bc830fbb800db1"
format_stamp: "Formatted at 2026-09-29 18:42:17 on dist-test-slave-smpn"
server_key: "068046f2dad53ed0e42a44acbdc2ebdf"
server_key_iv: "0535385d88cd3b95faf4bcda80028a85"
server_key_version: "encryptionkey@0"
I20260929 18:42:17.414229 9506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "0016b60f610a4161b1bc830fbb800db1"
format_stamp: "Formatted at 2026-09-29 18:42:17 on dist-test-slave-smpn"
server_key: "068046f2dad53ed0e42a44acbdc2ebdf"
server_key_iv: "0535385d88cd3b95faf4bcda80028a85"
server_key_version: "encryptionkey@0"
I20260929 18:42:17.417929 9506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 18:42:17.420313 9521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:17.421558 9506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:17.421687 9506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "0016b60f610a4161b1bc830fbb800db1"
format_stamp: "Formatted at 2026-09-29 18:42:17 on dist-test-slave-smpn"
server_key: "068046f2dad53ed0e42a44acbdc2ebdf"
server_key_iv: "0535385d88cd3b95faf4bcda80028a85"
server_key_version: "encryptionkey@0"
I20260929 18:42:17.421790 9506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:17.437173 9506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:17.439833 9506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:17.440011 9506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:17.447165 9506 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:37709
I20260929 18:42:17.447206 9583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:37709 every 8 connection(s)
I20260929 18:42:17.448071 9506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:17.450700 9584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:17.451700 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 9506
I20260929 18:42:17.451797 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:17.452026 7797 external_mini_cluster.cc:1428] Setting key 2caa6cd8f0ff14face006e8697e8c1f5
I20260929 18:42:17.456418 9584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1: Bootstrap starting.
I20260929 18:42:17.458578 9584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:17.459362 9584 log.cc:824] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:42:17 dist-test-slave-smpn krb5kdc[9493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:17.461202 9584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1: No bootstrap required, opened a new log
I20260929 18:42:17.464558 9584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0016b60f610a4161b1bc830fbb800db1" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37709 } }
I20260929 18:42:17.464784 9584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:17.464847 9584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0016b60f610a4161b1bc830fbb800db1, State: Initialized, Role: FOLLOWER
I20260929 18:42:17.464957 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:17.453368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40146 (local address 127.7.157.126:37709)
0929 18:42:17.453788 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:17.453803 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:17.453836 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:42:17.454461 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:17.454478 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:17.454808 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:17.455152 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:17.455915 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.456918 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.457752 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.458226 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.458375 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:17.460728 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:17.460733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:17.460746 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:17.460772 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:17.462319 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:17.462709 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:17.462714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:17.462718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:17.462823 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:17.463095 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:17.463098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:17.463100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:17.463476 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:17.463674 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:17.463926 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 18:42:17.464194 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":143,"threads_started":1}
I20260929 18:42:17.465306 9584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0016b60f610a4161b1bc830fbb800db1" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37709 } }
I20260929 18:42:17.465467 9584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:17.465528 9584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:17.465617 9584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:17.466513 9584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0016b60f610a4161b1bc830fbb800db1" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37709 } }
I20260929 18:42:17.466810 9584 leader_election.cc:302] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0016b60f610a4161b1bc830fbb800db1; no voters:
I20260929 18:42:17.467125 9584 leader_election.cc:288] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:17.467227 9589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:17.467424 9589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [term 1 LEADER]: Becoming Leader. State: Replica: 0016b60f610a4161b1bc830fbb800db1, State: Running, Role: LEADER
I20260929 18:42:17.467720 9589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0016b60f610a4161b1bc830fbb800db1" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37709 } }
I20260929 18:42:17.468235 9584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:17.469151 9591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0016b60f610a4161b1bc830fbb800db1. Latest consensus state: current_term: 1 leader_uuid: "0016b60f610a4161b1bc830fbb800db1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0016b60f610a4161b1bc830fbb800db1" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37709 } } }
I20260929 18:42:17.469269 9591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:17.469388 9590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0016b60f610a4161b1bc830fbb800db1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0016b60f610a4161b1bc830fbb800db1" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37709 } } }
I20260929 18:42:17.469537 9590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1 [sys.catalog]: This master's current role is: LEADER
W20260929 18:42:17.472806 9604 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:42:17.472873 9604 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:42:17.472922 9598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:17.473584 9598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:17.478865 9598 catalog_manager.cc:1347] Generated new cluster ID: b673b7d2486842b4ad1c4ca0b73a0b05
I20260929 18:42:17.478940 9598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:17.492468 9598 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:17.493466 9598 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:17.499780 9598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1: Generated new TSK 0
I20260929 18:42:17.500456 9598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:42:17.533667 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:37709
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:17.643042 9610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:17.643324 9610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:17.643388 9610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:17.647379 9610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:17.647432 9610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:17.647485 9610 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:42:17.647508 9610 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:42:17.647559 9610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:42:17.652768 9610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.157.126:37709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:17.653775 9610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:17.654786 9610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:17.664005 9616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:17.664013 9618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:17.664006 9615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:17.664217 9610 server_base.cc:1034] running on GCE node
I20260929 18:42:17.664810 9610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:17.665457 9610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:17.666622 9610 hybrid_clock.cc:648] HybridClock initialized: now 1790707337666596 us; error 35 us; skew 500 ppm
Sep 29 18:42:17 dist-test-slave-smpn krb5kdc[9493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707337, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:17.669374 9610 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:42:17.670317 9610 webserver.cc:460] Webserver started at http://127.7.157.65:37491/ using document root <none> and password file <none>
I20260929 18:42:17.670953 9610 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:17.671032 9610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:17.671262 9610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:17.673463 9610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "ef92c246c8b34f49a93d34818aa8eebc"
format_stamp: "Formatted at 2026-09-29 18:42:17 on dist-test-slave-smpn"
server_key: "2d5058eb3c2040f0952b464149c83ac7"
server_key_iv: "34a934b78ef21227ebe48488783588fa"
server_key_version: "encryptionkey@0"
I20260929 18:42:17.674000 9610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "ef92c246c8b34f49a93d34818aa8eebc"
format_stamp: "Formatted at 2026-09-29 18:42:17 on dist-test-slave-smpn"
server_key: "2d5058eb3c2040f0952b464149c83ac7"
server_key_iv: "34a934b78ef21227ebe48488783588fa"
server_key_version: "encryptionkey@0"
I20260929 18:42:17.677696 9610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:42:17.680100 9625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:17.681288 9610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:17.681391 9610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "ef92c246c8b34f49a93d34818aa8eebc"
format_stamp: "Formatted at 2026-09-29 18:42:17 on dist-test-slave-smpn"
server_key: "2d5058eb3c2040f0952b464149c83ac7"
server_key_iv: "34a934b78ef21227ebe48488783588fa"
server_key_version: "encryptionkey@0"
I20260929 18:42:17.681483 9610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:17.693179 9610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:17.695883 9610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:17.696053 9610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:17.697461 9610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:17.697510 9610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:17.697561 9610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:17.697588 9610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:42:17 dist-test-slave-smpn krb5kdc[9493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707337, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:17.707167 9610 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:39735
I20260929 18:42:17.707311 9740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:39735 every 8 connection(s)
I20260929 18:42:17.708177 9610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:42:17.709815 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 9610
I20260929 18:42:17.709915 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:42:17.710167 7797 external_mini_cluster.cc:1428] Setting key 077a72c1160a6adabf016c6b63e210ed
I20260929 18:42:17.711297 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:17.698231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:56539 (local address 127.7.157.126:37709)
0929 18:42:17.698335 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:17.698339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:17.699385 (+ 1046us) server_negotiation.cc:407] Connection header received
0929 18:42:17.700386 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:17.700389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:17.700445 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:17.700519 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:17.701929 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.702766 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.704441 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.704798 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.704918 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:17.707706 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:17.707711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:17.707713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:17.707743 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:17.709160 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:17.709646 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:17.709648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:17.709650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:17.709721 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:17.710057 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:17.710060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:17.710062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:17.710297 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:17.710382 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:17.710977 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 18:42:17.711140 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:42:17.712081 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:17.698772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:37709 (local address 127.7.157.65:56539)
0929 18:42:17.699255 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:17.699294 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:17.700185 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:17.700722 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:17.700731 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:17.701162 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:17.701759 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:17.701779 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.702914 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.702918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:17.704312 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:17.704323 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.704939 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.704942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:17.705084 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:17.705732 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:17.705753 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:17.707534 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:17.709263 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:17.709269 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:17.709282 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:17.709545 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:17.709809 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:17.709812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:17.709814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:17.709971 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:17.710431 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:17.710437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:17.710725 (+ 288us) client_negotiation.cc:769] Sending connection context
0929 18:42:17.710907 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 18:42:17.711247 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":133,"threads_started":1}
I20260929 18:42:17.713510 9741 heartbeater.cc:347] Connected to a master server at 127.7.157.126:37709
I20260929 18:42:17.713650 9741 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:17.714226 9741 heartbeater.cc:511] Master 127.7.157.126:37709 requested a full tablet report, sending...
I20260929 18:42:17.715833 9537 ts_manager.cc:194] Registered new tserver with Master: ef92c246c8b34f49a93d34818aa8eebc (127.7.157.65:39735)
I20260929 18:42:17.717039 9537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:56539
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
I20260929 18:42:17.722761 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:17.715619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:42273 (local address 127.7.157.126:37709)
0929 18:42:17.715723 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:17.715727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:17.715911 (+ 184us) server_negotiation.cc:407] Connection header received
0929 18:42:17.716087 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:17.716089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:17.716132 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:17.716203 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:17.716716 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.717460 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.718522 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.718851 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.718955 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:17.720069 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:17.720082 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:17.720085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:17.720112 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:17.721427 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:17.721794 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:17.721796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:17.721798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:17.721878 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:17.722137 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:17.722139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:17.722140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:17.722296 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:17.722365 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:17.722557 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 18:42:17.722649 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:42:17.722846 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:17.715533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:37709 (local address 127.7.157.65:42273)
0929 18:42:17.715828 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:17.715841 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:17.715986 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:17.716228 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:17.716231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:17.716435 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:17.716557 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:17.716564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.717592 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.717595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:17.718397 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:17.718407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.718974 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.718978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:17.719083 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:17.719608 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:17.719620 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:17.719936 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:17.721520 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:17.721522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:17.721524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:17.721720 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:17.721969 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:17.721972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:17.721974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:17.722058 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:17.722359 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:17.722360 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:17.722429 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 18:42:17.722501 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 18:42:17.722655 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:42:17.737360 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:37709
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:17.843525 9750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:17.843791 9750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:17.843842 9750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:17.847784 9750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:17.847847 9750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:17.847887 9750 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:42:17.847906 9750 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:42:17.847939 9750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:42:17.853124 9750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.157.126:37709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:17.854111 9750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:17.855088 9750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:17.864351 9756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:17.864456 9755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:17.864324 9758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:17.864450 9750 server_base.cc:1034] running on GCE node
I20260929 18:42:17.865084 9750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:17.865715 9750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:17.866858 9750 hybrid_clock.cc:648] HybridClock initialized: now 1790707337866830 us; error 41 us; skew 500 ppm
Sep 29 18:42:17 dist-test-slave-smpn krb5kdc[9493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707337, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:17.869549 9750 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:42:17.870370 9750 webserver.cc:460] Webserver started at http://127.7.157.66:44515/ using document root <none> and password file <none>
I20260929 18:42:17.870886 9750 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:17.870944 9750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:17.871160 9750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:17.873134 9750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "388b72481aa04578b149bc0f93ec33f0"
format_stamp: "Formatted at 2026-09-29 18:42:17 on dist-test-slave-smpn"
server_key: "7d3fa831559b0713e9a87e838d35efa6"
server_key_iv: "ebe6fe4952eb921f4bb4cfe280e055c9"
server_key_version: "encryptionkey@0"
I20260929 18:42:17.873649 9750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "388b72481aa04578b149bc0f93ec33f0"
format_stamp: "Formatted at 2026-09-29 18:42:17 on dist-test-slave-smpn"
server_key: "7d3fa831559b0713e9a87e838d35efa6"
server_key_iv: "ebe6fe4952eb921f4bb4cfe280e055c9"
server_key_version: "encryptionkey@0"
I20260929 18:42:17.877090 9750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:42:17.879281 9765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:17.880388 9750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:17.880483 9750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "388b72481aa04578b149bc0f93ec33f0"
format_stamp: "Formatted at 2026-09-29 18:42:17 on dist-test-slave-smpn"
server_key: "7d3fa831559b0713e9a87e838d35efa6"
server_key_iv: "ebe6fe4952eb921f4bb4cfe280e055c9"
server_key_version: "encryptionkey@0"
I20260929 18:42:17.880568 9750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:17.890544 9750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:17.893175 9750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:17.893340 9750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:17.894737 9750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:17.894784 9750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:17.894835 9750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:17.894865 9750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:42:17 dist-test-slave-smpn krb5kdc[9493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707337, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:17.905535 9750 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:38575
I20260929 18:42:17.905565 9880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:38575 every 8 connection(s)
I20260929 18:42:17.906493 9750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:42:17.908020 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:17.895511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:33755 (local address 127.7.157.126:37709)
0929 18:42:17.895671 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:17.895677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:17.897769 (+ 2092us) server_negotiation.cc:407] Connection header received
0929 18:42:17.897833 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:17.897836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:17.897895 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:17.897993 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:17.899207 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.900010 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.901237 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.901565 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.901692 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:17.904555 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:17.904559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:17.904562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:17.904591 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:17.905965 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:17.906463 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:17.906465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:17.906466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:17.906534 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:17.906918 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:17.906922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:17.906924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:17.907184 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:17.907279 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:17.907785 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 18:42:17.907884 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:42:17.908735 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:17.895926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:37709 (local address 127.7.157.66:33755)
0929 18:42:17.896375 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:17.896422 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:17.897482 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:17.898172 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:17.898181 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:17.898511 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:17.899041 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:17.899055 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.900129 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.900133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:17.901133 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:17.901143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.901753 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.901758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:17.901954 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:17.902558 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:17.902574 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:17.904369 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:17.906066 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:17.906071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:17.906085 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:17.906380 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:17.906654 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:17.906657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:17.906659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:17.906830 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:17.907287 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:17.907293 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:17.907568 (+ 275us) client_negotiation.cc:769] Sending connection context
0929 18:42:17.907728 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 18:42:17.907972 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
I20260929 18:42:17.909761 9881 heartbeater.cc:347] Connected to a master server at 127.7.157.126:37709
I20260929 18:42:17.909893 9881 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:17.910403 9881 heartbeater.cc:511] Master 127.7.157.126:37709 requested a full tablet report, sending...
I20260929 18:42:17.911451 9538 ts_manager.cc:194] Registered new tserver with Master: 388b72481aa04578b149bc0f93ec33f0 (127.7.157.66:38575)
I20260929 18:42:17.912163 9538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:33755
I20260929 18:42:17.912735 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 9750
I20260929 18:42:17.912839 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:42:17.913085 7797 external_mini_cluster.cc:1428] Setting key 5715821b7fb12d39c38254a9a71fc58c
I20260929 18:42:17.919054 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:17.911945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:50279 (local address 127.7.157.126:37709)
0929 18:42:17.912047 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:17.912050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:17.912264 (+ 214us) server_negotiation.cc:407] Connection header received
0929 18:42:17.912357 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:17.912359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:17.912400 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:17.912470 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:17.913251 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.913917 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.914871 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.915190 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.915283 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:17.916283 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:17.916286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:17.916287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:17.916307 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:17.917611 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:17.917975 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:17.917979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:17.917981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:17.918064 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:17.918292 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:17.918295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:17.918296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:17.918491 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:17.918574 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:17.918815 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 18:42:17.918913 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:42:17.919104 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:17.911880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:37709 (local address 127.7.157.66:50279)
0929 18:42:17.912187 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:17.912201 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:17.912287 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:17.912707 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:17.912710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:17.912952 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:17.913108 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:17.913115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.914201 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.914204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:17.914772 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:17.914781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:17.915280 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:17.915282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:17.915346 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:17.915820 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:17.915834 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:17.916165 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:17.917715 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:17.917718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:17.917719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:17.917898 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:17.918137 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:17.918139 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:17.918141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:17.918203 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:17.918587 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:17.918591 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:17.918673 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 18:42:17.918756 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:42:17.918889 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:42:17.940426 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:37709
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:18.049682 9890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:18.049950 9890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:18.049999 9890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:18.053983 9890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:18.054047 9890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:18.054090 9890 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:42:18.054116 9890 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:42:18.054167 9890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:42:18.059429 9890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.157.126:37709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:18.060508 9890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:18.061573 9890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:18.071061 9895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:18.071146 9898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:18.071184 9890 server_base.cc:1034] running on GCE node
W20260929 18:42:18.071146 9896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:18.071730 9890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:18.072376 9890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:18.073585 9890 hybrid_clock.cc:648] HybridClock initialized: now 1790707338073559 us; error 33 us; skew 500 ppm
Sep 29 18:42:18 dist-test-slave-smpn krb5kdc[9493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707338, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:18.076391 9890 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:42:18.077296 9890 webserver.cc:460] Webserver started at http://127.7.157.67:37933/ using document root <none> and password file <none>
I20260929 18:42:18.077852 9890 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:18.077912 9890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:18.078104 9890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:18.080235 9890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "1c92df9ae320465d90fbf8ceb89a66c2"
format_stamp: "Formatted at 2026-09-29 18:42:18 on dist-test-slave-smpn"
server_key: "1780148796ede8de5110d624b86ee720"
server_key_iv: "442cfaa5da74d283f77917891a7b3b8c"
server_key_version: "encryptionkey@0"
I20260929 18:42:18.080802 9890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "1c92df9ae320465d90fbf8ceb89a66c2"
format_stamp: "Formatted at 2026-09-29 18:42:18 on dist-test-slave-smpn"
server_key: "1780148796ede8de5110d624b86ee720"
server_key_iv: "442cfaa5da74d283f77917891a7b3b8c"
server_key_version: "encryptionkey@0"
I20260929 18:42:18.084627 9890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:42:18.087149 9905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:18.088343 9890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:18.088441 9890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "1c92df9ae320465d90fbf8ceb89a66c2"
format_stamp: "Formatted at 2026-09-29 18:42:18 on dist-test-slave-smpn"
server_key: "1780148796ede8de5110d624b86ee720"
server_key_iv: "442cfaa5da74d283f77917891a7b3b8c"
server_key_version: "encryptionkey@0"
I20260929 18:42:18.088538 9890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:18.101792 9890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:18.104511 9890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:18.104689 9890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:18.106101 9890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:18.106161 9890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:18.106213 9890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:18.106241 9890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:42:18 dist-test-slave-smpn krb5kdc[9493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707338, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:18.116443 9890 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:33521
I20260929 18:42:18.116459 10020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:33521 every 8 connection(s)
I20260929 18:42:18.117301 9890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:42:18.118637 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.106813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:51197 (local address 127.7.157.126:37709)
0929 18:42:18.106965 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.106971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.107866 (+ 895us) server_negotiation.cc:407] Connection header received
0929 18:42:18.109044 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.109047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.109121 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.109213 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:18.110442 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.110967 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.112184 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.112398 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.112483 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.115241 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:18.115244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:18.115247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:18.115275 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:18.116671 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:18.117124 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:18.117128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:18.117130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:18.117208 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:18.117537 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:18.117540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:18.117542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:18.117756 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:18.117845 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.118396 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.118511 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:42:18.119379 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.107219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:37709 (local address 127.7.157.67:51197)
0929 18:42:18.107689 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:18.107747 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:18.108796 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:18.109391 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:18.109398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:18.109745 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:18.110306 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.110322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.111105 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.111109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.112068 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.112078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.112544 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.112548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.112713 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.113366 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:18.113387 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:18.115054 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:18.116769 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:18.116775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:18.116789 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:18.117022 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:18.117284 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:18.117287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:18.117289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:18.117455 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:18.117893 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:18.117899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:18.118156 (+ 257us) client_negotiation.cc:769] Sending connection context
0929 18:42:18.118302 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 18:42:18.118535 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":100,"threads_started":1}
I20260929 18:42:18.120344 10021 heartbeater.cc:347] Connected to a master server at 127.7.157.126:37709
I20260929 18:42:18.120467 10021 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:18.121029 10021 heartbeater.cc:511] Master 127.7.157.126:37709 requested a full tablet report, sending...
I20260929 18:42:18.122035 9537 ts_manager.cc:194] Registered new tserver with Master: 1c92df9ae320465d90fbf8ceb89a66c2 (127.7.157.67:33521)
I20260929 18:42:18.122748 9537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:51197
I20260929 18:42:18.126562 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 9890
I20260929 18:42:18.126660 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:42:18.126986 7797 external_mini_cluster.cc:1428] Setting key 3daa3eadbcc7c2f47b3afc0e9244cd0a
I20260929 18:42:18.128870 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.122525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:41329 (local address 127.7.157.126:37709)
0929 18:42:18.122634 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.122639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.122763 (+ 124us) server_negotiation.cc:407] Connection header received
0929 18:42:18.122863 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.122866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.122908 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.122969 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:18.123420 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.123904 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.124681 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.124940 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.125020 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.125995 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:18.125998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:18.126001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:18.126023 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:18.127435 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:18.127851 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:18.127853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:18.127855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:18.127923 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:18.128169 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:18.128172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:18.128173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:18.128326 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:18.128406 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.128651 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.128744 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:42:18.128868 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:42:18.128881 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.122464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:37709 (local address 127.7.157.67:41329)
0929 18:42:18.122693 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:18.122704 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:18.122789 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:18.122991 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:18.122994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:18.123209 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:18.123311 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.123318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.124020 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.124023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.124572 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.124579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.125045 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.125048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.125135 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.125569 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:18.125582 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:18.125872 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:18.127524 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:18.127527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:18.127529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:18.127761 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:18.127998 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:18.128001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:18.128003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:18.128085 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:18.128415 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:18.128418 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:18.128499 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 18:42:18.128580 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:42:18.128707 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":75,"threads_started":1}
Sep 29 18:42:18 dist-test-slave-smpn krb5kdc[9493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:42:18 dist-test-slave-smpn krb5kdc[9493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:18.152040 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.144770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40152 (local address 127.7.157.126:37709)
0929 18:42:18.144883 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.144886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.144897 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:42:18.144934 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.144936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.144973 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.145037 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:18.145722 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.146252 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.147047 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.147310 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.147398 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.149494 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:18.149497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:18.149499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:18.149523 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:18.150546 (+ 1023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:18.150926 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:18.150930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:18.150932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:18.151004 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:18.151273 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:18.151276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:18.151278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:18.151450 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:18.151568 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.151781 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.151889 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:42:18.154760 9537 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:42:18.157770 9537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:18.171185 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.166360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:39735 (local address 127.0.0.1:57540)
0929 18:42:18.166830 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:18.166843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:18.166908 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:18.167583 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:18.167585 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:18.167600 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:18.167738 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.167745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.168696 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.168700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.169835 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.169843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.170650 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.170654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.170754 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.170789 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 18:42:18.170892 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:42:18.171010 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":26,"thread_start_us":96,"threads_started":1}
I20260929 18:42:18.171197 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.166770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.7.157.65:39735)
0929 18:42:18.167157 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.167166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.167183 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:42:18.167240 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.167244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.167416 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.167597 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:18.167851 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.168601 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.169937 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.170548 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.170642 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.170801 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.170925 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.171013 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":83,"threads_started":1}
I20260929 18:42:18.171510 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.165627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:33521 (local address 127.0.0.1:58262)
0929 18:42:18.166449 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:18.166500 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:18.166631 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:18.167381 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:18.167384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:18.167420 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:18.167640 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.167648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.169127 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.169130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.170069 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.170078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.171127 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.171130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.171249 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.171271 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:42:18.171321 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 18:42:18.171379 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":56,"spinlock_wait_cycles":6400,"thread_start_us":86,"threads_started":1}
I20260929 18:42:18.172079 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.165748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.7.157.67:33521)
0929 18:42:18.166061 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.166074 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.166884 (+ 810us) server_negotiation.cc:407] Connection header received
0929 18:42:18.166972 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.166978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.167224 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.167515 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:18.167751 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.169001 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.170194 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.170973 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.171672 (+ 699us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.171823 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.171870 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.171948 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":112,"threads_started":1}
I20260929 18:42:18.172812 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.165664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:38575 (local address 127.0.0.1:47536)
0929 18:42:18.166632 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:18.166645 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:18.166729 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:18.167379 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:18.167384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:18.167418 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:18.167819 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.167825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.169154 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.169158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.171228 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.171237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.172441 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.172445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.172537 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.172558 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:42:18.172636 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:42:18.172697 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":78,"spinlock_wait_cycles":7936,"thread_start_us":85,"threads_started":1}
I20260929 18:42:18.173421 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.166273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.7.157.66:38575)
0929 18:42:18.166953 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.166964 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.166982 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:42:18.167038 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.167043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.167224 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.167381 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:18.167914 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.169041 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.171564 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.172334 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.172431 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.172574 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.172660 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.172747 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":111,"threads_started":1}
I20260929 18:42:18.174173 9675 tablet_service.cc:1467] Processing CreateTablet for tablet 6d1c452bd5a34b4a82704e170956c4a9 (DEFAULT_TABLE table=test-table [id=2c40585dcd2e4a42b3618aeea9fcf172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:18.175024 9675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d1c452bd5a34b4a82704e170956c4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:18.175208 9955 tablet_service.cc:1467] Processing CreateTablet for tablet 6d1c452bd5a34b4a82704e170956c4a9 (DEFAULT_TABLE table=test-table [id=2c40585dcd2e4a42b3618aeea9fcf172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:18.176258 9955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d1c452bd5a34b4a82704e170956c4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:18.175918 9815 tablet_service.cc:1467] Processing CreateTablet for tablet 6d1c452bd5a34b4a82704e170956c4a9 (DEFAULT_TABLE table=test-table [id=2c40585dcd2e4a42b3618aeea9fcf172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:18.176860 9815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d1c452bd5a34b4a82704e170956c4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:18.181674 10044 tablet_bootstrap.cc:492] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc: Bootstrap starting.
I20260929 18:42:18.183553 10045 tablet_bootstrap.cc:492] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2: Bootstrap starting.
I20260929 18:42:18.184059 10044 tablet_bootstrap.cc:654] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:18.184938 10046 tablet_bootstrap.cc:492] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0: Bootstrap starting.
I20260929 18:42:18.185411 10044 log.cc:824] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:18.185668 10045 tablet_bootstrap.cc:654] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:18.186442 10045 log.cc:824] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:18.186807 10046 tablet_bootstrap.cc:654] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:18.187544 10044 tablet_bootstrap.cc:492] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc: No bootstrap required, opened a new log
I20260929 18:42:18.187599 10046 log.cc:824] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:18.187747 10044 ts_tablet_manager.cc:1397] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:42:18.188059 10045 tablet_bootstrap.cc:492] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2: No bootstrap required, opened a new log
I20260929 18:42:18.188246 10045 ts_tablet_manager.cc:1397] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:42:18.189529 10046 tablet_bootstrap.cc:492] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0: No bootstrap required, opened a new log
I20260929 18:42:18.189707 10046 ts_tablet_manager.cc:1397] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:42:18.191470 10045 raft_consensus.cc:348] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } }
I20260929 18:42:18.191483 10044 raft_consensus.cc:348] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } }
I20260929 18:42:18.191702 10045 raft_consensus.cc:374] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:18.191741 10044 raft_consensus.cc:374] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:18.191752 10045 raft_consensus.cc:729] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c92df9ae320465d90fbf8ceb89a66c2, State: Initialized, Role: FOLLOWER
I20260929 18:42:18.191789 10044 raft_consensus.cc:729] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef92c246c8b34f49a93d34818aa8eebc, State: Initialized, Role: FOLLOWER
I20260929 18:42:18.192160 10045 consensus_queue.cc:260] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } }
I20260929 18:42:18.192198 10044 consensus_queue.cc:260] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } }
I20260929 18:42:18.193040 10045 ts_tablet_manager.cc:1428] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260929 18:42:18.193120 10044 ts_tablet_manager.cc:1428] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:42:18.193391 10021 heartbeater.cc:503] Master 127.7.157.126:37709 was elected leader, sending a full tablet report...
I20260929 18:42:18.193429 9741 heartbeater.cc:503] Master 127.7.157.126:37709 was elected leader, sending a full tablet report...
I20260929 18:42:18.194556 10046 raft_consensus.cc:348] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } }
I20260929 18:42:18.194818 10046 raft_consensus.cc:374] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:18.194864 10046 raft_consensus.cc:729] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 388b72481aa04578b149bc0f93ec33f0, State: Initialized, Role: FOLLOWER
I20260929 18:42:18.195292 10046 consensus_queue.cc:260] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } }
I20260929 18:42:18.195930 9881 heartbeater.cc:503] Master 127.7.157.126:37709 was elected leader, sending a full tablet report...
I20260929 18:42:18.196247 10046 ts_tablet_manager.cc:1428] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:42:18.208873 10050 raft_consensus.cc:482] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:18.209066 10050 raft_consensus.cc:504] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } }
W20260929 18:42:18.209829 9742 tablet.cc:2369] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:42:18.210752 10050 leader_election.cc:288] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 388b72481aa04578b149bc0f93ec33f0 (127.7.157.66:38575), ef92c246c8b34f49a93d34818aa8eebc (127.7.157.65:39735)
I20260929 18:42:18.213922 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.210545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:38575 (local address 127.7.157.67:33229)
0929 18:42:18.210672 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:18.210688 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:18.210768 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:18.211030 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:18.211033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:18.211053 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:18.211193 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.211201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.211943 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.211946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.212756 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.212766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.213594 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.213598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.213689 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.213709 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:42:18.213768 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 18:42:18.213821 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 18:42:18.213979 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.210645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:33229 (local address 127.7.157.66:38575)
0929 18:42:18.210752 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.210757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.210773 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:42:18.210878 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.210881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.210930 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.210996 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:18.211294 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.211841 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.212881 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.213475 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.213605 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.213657 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.213795 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.213860 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:42:18.214591 9829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d1c452bd5a34b4a82704e170956c4a9" candidate_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "388b72481aa04578b149bc0f93ec33f0" is_pre_election: true
I20260929 18:42:18.214931 9829 raft_consensus.cc:2454] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c92df9ae320465d90fbf8ceb89a66c2 in term 0.
I20260929 18:42:18.215166 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.210966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:39735 (local address 127.7.157.67:53833)
0929 18:42:18.211346 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:18.211358 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:18.211437 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:18.211637 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:18.211640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:18.211655 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:18.211780 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.211785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.212449 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.212452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.213999 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.214009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.214787 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.214791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.214869 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.214887 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:42:18.214957 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:42:18.215007 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":176,"threads_started":1}
I20260929 18:42:18.215174 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.211045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:53833 (local address 127.7.157.65:39735)
0929 18:42:18.211211 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.211216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.211426 (+ 210us) server_negotiation.cc:407] Connection header received
0929 18:42:18.211515 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.211518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.211558 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.211623 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:18.211876 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.212355 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.214153 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.214661 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.214773 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.214816 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.214979 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.215047 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 18:42:18.215468 9909 leader_election.cc:302] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c92df9ae320465d90fbf8ceb89a66c2, 388b72481aa04578b149bc0f93ec33f0; no voters:
I20260929 18:42:18.215662 9695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d1c452bd5a34b4a82704e170956c4a9" candidate_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef92c246c8b34f49a93d34818aa8eebc" is_pre_election: true
I20260929 18:42:18.215726 10050 raft_consensus.cc:2781] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:18.215801 10050 raft_consensus.cc:482] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:18.215842 10050 raft_consensus.cc:3037] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:18.215930 9695 raft_consensus.cc:2454] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c92df9ae320465d90fbf8ceb89a66c2 in term 0.
I20260929 18:42:18.216943 10050 raft_consensus.cc:504] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } }
I20260929 18:42:18.217222 10050 leader_election.cc:288] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [CANDIDATE]: Term 1 election: Requested vote from peers 388b72481aa04578b149bc0f93ec33f0 (127.7.157.66:38575), ef92c246c8b34f49a93d34818aa8eebc (127.7.157.65:39735)
I20260929 18:42:18.217525 9695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d1c452bd5a34b4a82704e170956c4a9" candidate_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef92c246c8b34f49a93d34818aa8eebc"
I20260929 18:42:18.217506 9829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d1c452bd5a34b4a82704e170956c4a9" candidate_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "388b72481aa04578b149bc0f93ec33f0"
I20260929 18:42:18.217628 9829 raft_consensus.cc:3037] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:18.217658 9695 raft_consensus.cc:3037] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:18.218812 9829 raft_consensus.cc:2454] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c92df9ae320465d90fbf8ceb89a66c2 in term 1.
I20260929 18:42:18.218812 9695 raft_consensus.cc:2454] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c92df9ae320465d90fbf8ceb89a66c2 in term 1.
I20260929 18:42:18.219125 9908 leader_election.cc:302] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c92df9ae320465d90fbf8ceb89a66c2, ef92c246c8b34f49a93d34818aa8eebc; no voters:
I20260929 18:42:18.219295 10050 raft_consensus.cc:2781] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:18.219471 10050 raft_consensus.cc:686] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c92df9ae320465d90fbf8ceb89a66c2, State: Running, Role: LEADER
I20260929 18:42:18.219758 10050 consensus_queue.cc:237] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } }
I20260929 18:42:18.222771 9537 catalog_manager.cc:5495] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c92df9ae320465d90fbf8ceb89a66c2 (127.7.157.67). New cstate: current_term: 1 leader_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } health_report { overall_health: UNKNOWN } } }
Sep 29 18:42:18 dist-test-slave-smpn krb5kdc[9493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707337, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:18.259925 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.252766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40160 (local address 127.7.157.126:37709)
0929 18:42:18.252890 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.252895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.253046 (+ 151us) server_negotiation.cc:407] Connection header received
0929 18:42:18.253150 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.253152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.253191 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.253255 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:18.253858 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.254347 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.255041 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.255273 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.255340 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.257361 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:18.257364 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:18.257366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:18.257384 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:18.258422 (+ 1038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:18.258816 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:18.258820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:18.258821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:18.258885 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:18.259139 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:18.259142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:18.259144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:18.259302 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:18.259382 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.259657 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.259790 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":47}
I20260929 18:42:18.259925 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.252705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:37709 (local address 127.0.0.1:40160)
0929 18:42:18.252966 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:18.252979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:18.253080 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:18.253283 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:18.253285 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:18.253684 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:18.253774 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.253779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.254418 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.254420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.254967 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.254973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.255355 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.255357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.255417 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.255937 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:18.255949 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:18.257247 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:18.258511 (+ 1264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:18.258520 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:18.258526 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:18.258733 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:18.258959 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:18.258962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:18.258964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:18.259032 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:18.259394 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:18.259402 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:18.259501 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 18:42:18.259591 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:42:18.259739 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":100,"threads_started":1}
I20260929 18:42:18.263736 9537 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:40160:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:42:18.264402 9537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:18.268939 9675 tablet_service.cc:1467] Processing CreateTablet for tablet 22ddb658bc75438f906e663344727b78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9aa2ab1c1fd404c8f8b00a1ca1ef377]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:42:18.269124 9815 tablet_service.cc:1467] Processing CreateTablet for tablet 22ddb658bc75438f906e663344727b78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9aa2ab1c1fd404c8f8b00a1ca1ef377]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:42:18.269052 9955 tablet_service.cc:1467] Processing CreateTablet for tablet 22ddb658bc75438f906e663344727b78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9aa2ab1c1fd404c8f8b00a1ca1ef377]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:42:18.269708 9675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22ddb658bc75438f906e663344727b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:18.269708 9815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22ddb658bc75438f906e663344727b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:18.269713 9955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22ddb658bc75438f906e663344727b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:18.272353 10045 tablet_bootstrap.cc:492] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2: Bootstrap starting.
I20260929 18:42:18.272419 10046 tablet_bootstrap.cc:492] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0: Bootstrap starting.
I20260929 18:42:18.272449 10044 tablet_bootstrap.cc:492] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc: Bootstrap starting.
I20260929 18:42:18.273164 10044 tablet_bootstrap.cc:654] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:18.273319 10045 tablet_bootstrap.cc:654] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:18.273351 10046 tablet_bootstrap.cc:654] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:18.274271 10044 tablet_bootstrap.cc:492] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc: No bootstrap required, opened a new log
I20260929 18:42:18.274353 10044 ts_tablet_manager.cc:1397] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:18.274813 10044 raft_consensus.cc:348] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.274940 10044 raft_consensus.cc:374] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:18.274973 10044 raft_consensus.cc:729] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef92c246c8b34f49a93d34818aa8eebc, State: Initialized, Role: FOLLOWER
I20260929 18:42:18.275142 10046 tablet_bootstrap.cc:492] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0: No bootstrap required, opened a new log
I20260929 18:42:18.275130 10044 consensus_queue.cc:260] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.275213 10046 ts_tablet_manager.cc:1397] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:42:18.275393 10044 ts_tablet_manager.cc:1428] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:18.275417 10045 tablet_bootstrap.cc:492] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2: No bootstrap required, opened a new log
I20260929 18:42:18.275493 10045 ts_tablet_manager.cc:1397] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 18:42:18.275386 10051 tablet_replica.cc:439] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } }
I20260929 18:42:18.275563 10051 tablet_replica.cc:442] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:18.275631 10046 raft_consensus.cc:348] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.275727 10046 raft_consensus.cc:374] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:18.275755 10046 raft_consensus.cc:729] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 388b72481aa04578b149bc0f93ec33f0, State: Initialized, Role: FOLLOWER
I20260929 18:42:18.275859 10046 consensus_queue.cc:260] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.275905 10045 raft_consensus.cc:348] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.276012 10045 raft_consensus.cc:374] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:18.276044 10045 raft_consensus.cc:729] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c92df9ae320465d90fbf8ceb89a66c2, State: Initialized, Role: FOLLOWER
I20260929 18:42:18.276108 10046 ts_tablet_manager.cc:1428] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:42:18.276099 10052 tablet_replica.cc:439] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } }
I20260929 18:42:18.276154 10045 consensus_queue.cc:260] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.276229 10052 tablet_replica.cc:442] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:18.276379 10050 tablet_replica.cc:439] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } }
I20260929 18:42:18.276495 10050 tablet_replica.cc:442] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:18.276391 10045 ts_tablet_manager.cc:1428] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:18.333786 10051 raft_consensus.cc:482] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:18.333922 10051 raft_consensus.cc:504] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.335048 10051 leader_election.cc:288] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 388b72481aa04578b149bc0f93ec33f0 (127.7.157.66:38575), 1c92df9ae320465d90fbf8ceb89a66c2 (127.7.157.67:33521)
I20260929 18:42:18.338591 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.335328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:38575 (local address 127.7.157.65:57927)
0929 18:42:18.335605 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:18.335616 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:18.335720 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:18.335948 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:18.335951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:18.335965 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:18.336103 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.336108 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.336784 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.336787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.337572 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.337580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.338182 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.338186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.338280 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.338298 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:42:18.338360 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:42:18.338428 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":2,"spinlock_wait_cycles":5504,"thread_start_us":88,"threads_started":1}
I20260929 18:42:18.338567 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.335425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:57927 (local address 127.7.157.66:38575)
0929 18:42:18.335546 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.335551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.335684 (+ 133us) server_negotiation.cc:407] Connection header received
0929 18:42:18.335814 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.335817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.335858 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.335925 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:18.336196 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.336685 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.337682 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.338057 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.338166 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.338221 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.338383 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.338449 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:42:18.338857 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.335446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:38333 (local address 127.7.157.67:33521)
0929 18:42:18.335644 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.335649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.335729 (+ 80us) server_negotiation.cc:407] Connection header received
0929 18:42:18.335854 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.335856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.335894 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.335954 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:18.336560 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.337041 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.337967 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.338353 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.338453 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.338500 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.338652 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.338734 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 18:42:18.338951 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.335328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:33521 (local address 127.7.157.65:38333)
0929 18:42:18.335644 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:18.335657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:18.335759 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:18.335970 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:18.335972 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:18.336011 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:18.336449 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.336455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.337144 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.337147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.337852 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.337859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.338453 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.338457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.338543 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.338560 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:42:18.338621 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 18:42:18.338815 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":53,"spinlock_wait_cycles":2560,"thread_start_us":64,"threads_started":1}
I20260929 18:42:18.339270 9829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22ddb658bc75438f906e663344727b78" candidate_uuid: "ef92c246c8b34f49a93d34818aa8eebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "388b72481aa04578b149bc0f93ec33f0" is_pre_election: true
I20260929 18:42:18.339421 9829 raft_consensus.cc:2454] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef92c246c8b34f49a93d34818aa8eebc in term 0.
I20260929 18:42:18.339469 9975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22ddb658bc75438f906e663344727b78" candidate_uuid: "ef92c246c8b34f49a93d34818aa8eebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" is_pre_election: true
I20260929 18:42:18.339712 9975 raft_consensus.cc:2454] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef92c246c8b34f49a93d34818aa8eebc in term 0.
I20260929 18:42:18.339782 9629 leader_election.cc:302] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 388b72481aa04578b149bc0f93ec33f0, ef92c246c8b34f49a93d34818aa8eebc; no voters:
I20260929 18:42:18.340060 10051 raft_consensus.cc:2781] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:18.340122 10051 raft_consensus.cc:482] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:18.340149 10051 raft_consensus.cc:3037] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:18.341081 10051 raft_consensus.cc:504] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.341391 10051 leader_election.cc:288] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [CANDIDATE]: Term 1 election: Requested vote from peers 388b72481aa04578b149bc0f93ec33f0 (127.7.157.66:38575), 1c92df9ae320465d90fbf8ceb89a66c2 (127.7.157.67:33521)
I20260929 18:42:18.341691 9975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22ddb658bc75438f906e663344727b78" candidate_uuid: "ef92c246c8b34f49a93d34818aa8eebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c92df9ae320465d90fbf8ceb89a66c2"
I20260929 18:42:18.341710 9829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22ddb658bc75438f906e663344727b78" candidate_uuid: "ef92c246c8b34f49a93d34818aa8eebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "388b72481aa04578b149bc0f93ec33f0"
I20260929 18:42:18.341809 9829 raft_consensus.cc:3037] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:18.341862 9975 raft_consensus.cc:3037] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:18.342906 9829 raft_consensus.cc:2454] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef92c246c8b34f49a93d34818aa8eebc in term 1.
I20260929 18:42:18.342906 9975 raft_consensus.cc:2454] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef92c246c8b34f49a93d34818aa8eebc in term 1.
I20260929 18:42:18.343233 9626 leader_election.cc:302] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c92df9ae320465d90fbf8ceb89a66c2, ef92c246c8b34f49a93d34818aa8eebc; no voters:
I20260929 18:42:18.343472 10051 raft_consensus.cc:2781] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:18.343693 10051 raft_consensus.cc:686] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [term 1 LEADER]: Becoming Leader. State: Replica: ef92c246c8b34f49a93d34818aa8eebc, State: Running, Role: LEADER
I20260929 18:42:18.344075 10051 consensus_queue.cc:237] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.346091 10067 tablet_replica.cc:439] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc: TxnStatusTablet state changed. Reason: New leader ef92c246c8b34f49a93d34818aa8eebc. Latest consensus state: current_term: 1 leader_uuid: "ef92c246c8b34f49a93d34818aa8eebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } }
I20260929 18:42:18.346273 10067 tablet_replica.cc:442] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:42:18.346563 10069 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:42:18.346756 10069 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:42:18.347389 9537 catalog_manager.cc:5495] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc reported cstate change: term changed from 0 to 1, leader changed from <none> to ef92c246c8b34f49a93d34818aa8eebc (127.7.157.65). New cstate: current_term: 1 leader_uuid: "ef92c246c8b34f49a93d34818aa8eebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } health_report { overall_health: UNKNOWN } } }
W20260929 18:42:18.368232 10022 tablet.cc:2369] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:42:18.375385 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.371810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57546 (local address 127.7.157.65:39735)
0929 18:42:18.371945 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.371950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.372021 (+ 71us) server_negotiation.cc:407] Connection header received
0929 18:42:18.372236 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.372239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.372292 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.372374 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:18.372620 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.373102 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.373925 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.374137 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.374220 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.374524 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:18.374963 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:18.375090 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.375189 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.375241 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:42:18.375385 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.371697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:39735 (local address 127.0.0.1:57546)
0929 18:42:18.371962 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:18.371976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:18.372088 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:18.372404 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:18.372406 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:18.372413 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:42:18.372517 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.372522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.373210 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.373214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.373816 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.373824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.374238 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.374242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.374340 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.374384 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:42:18.375087 (+ 703us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:42:18.375114 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 18:42:18.375178 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:42:18.375268 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260929 18:42:18.381407 9975 raft_consensus.cc:1261] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Refusing update from remote peer ef92c246c8b34f49a93d34818aa8eebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:18.381505 9829 raft_consensus.cc:1261] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0 [term 1 FOLLOWER]: Refusing update from remote peer ef92c246c8b34f49a93d34818aa8eebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:18.382052 10051 consensus_queue.cc:1035] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:18.382272 10067 consensus_queue.cc:1035] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:18.385916 10050 tablet_replica.cc:439] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2: TxnStatusTablet state changed. Reason: New leader ef92c246c8b34f49a93d34818aa8eebc. Latest consensus state: current_term: 1 leader_uuid: "ef92c246c8b34f49a93d34818aa8eebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } }
I20260929 18:42:18.386068 10050 tablet_replica.cc:442] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:18.386649 10052 tablet_replica.cc:439] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0: TxnStatusTablet state changed. Reason: New leader ef92c246c8b34f49a93d34818aa8eebc. Latest consensus state: current_term: 1 leader_uuid: "ef92c246c8b34f49a93d34818aa8eebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } }
I20260929 18:42:18.386770 10052 tablet_replica.cc:442] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:18.389232 10051 tablet_replica.cc:439] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef92c246c8b34f49a93d34818aa8eebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } }
I20260929 18:42:18.389360 10051 tablet_replica.cc:442] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:42:18.389631 10052 tablet_replica.cc:439] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef92c246c8b34f49a93d34818aa8eebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } }
I20260929 18:42:18.389737 10052 tablet_replica.cc:442] T 22ddb658bc75438f906e663344727b78 P 388b72481aa04578b149bc0f93ec33f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:18.389835 10072 mvcc.cc:205] Tried to move back new op lower bound from 7334737258003054592 to 7334737257860014080. Current Snapshot: MvccSnapshot[applied={T|T < 7334737258003054592}]
I20260929 18:42:18.390745 10074 mvcc.cc:205] Tried to move back new op lower bound from 7334737258003054592 to 7334737257860014080. Current Snapshot: MvccSnapshot[applied={T|T < 7334737258003054592}]
I20260929 18:42:18.392721 10051 tablet_replica.cc:439] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef92c246c8b34f49a93d34818aa8eebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } }
I20260929 18:42:18.392831 10051 tablet_replica.cc:442] T 22ddb658bc75438f906e663344727b78 P ef92c246c8b34f49a93d34818aa8eebc: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:42:18.393765 10073 mvcc.cc:205] Tried to move back new op lower bound from 7334737258003054592 to 7334737257860014080. Current Snapshot: MvccSnapshot[applied={T|T < 7334737258003054592}]
I20260929 18:42:18.393944 10050 tablet_replica.cc:439] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef92c246c8b34f49a93d34818aa8eebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } } }
I20260929 18:42:18.394047 10050 tablet_replica.cc:442] T 22ddb658bc75438f906e663344727b78 P 1c92df9ae320465d90fbf8ceb89a66c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:18.404034 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.400142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58264 (local address 127.7.157.67:33521)
0929 18:42:18.400259 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.400264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.400279 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:42:18.400393 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.400396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.400457 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.400561 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:18.400751 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.401513 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.402457 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.402806 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.402907 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.403126 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:18.403686 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:18.403799 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.403835 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.403890 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 18:42:18.408269 9882 tablet.cc:2369] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:42:18.414742 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.411665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:50617 (local address 127.7.157.65:39735)
0929 18:42:18.411782 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.411787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.411829 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:42:18.411990 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.411993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.412037 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.412096 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:18.412337 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.412830 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.413713 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.413924 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.414007 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.414322 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:18.414395 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:18.414455 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.414543 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.414592 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:42:18.414742 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.411544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:39735 (local address 127.7.157.67:50617)
0929 18:42:18.411769 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:18.411784 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:18.411878 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:18.412123 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:18.412126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:18.412133 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:42:18.412246 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.412251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.412928 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.412931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.413586 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.413594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.414023 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.414027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.414126 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.414175 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:42:18.414471 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:42:18.414477 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:42:18.414533 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:42:18.414612 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260929 18:42:18.421849 9835 raft_consensus.cc:1261] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c92df9ae320465d90fbf8ceb89a66c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:18.421948 9695 raft_consensus.cc:1261] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc [term 1 FOLLOWER]: Refusing update from remote peer 1c92df9ae320465d90fbf8ceb89a66c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:18.422365 10050 consensus_queue.cc:1035] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:18.422667 10050 consensus_queue.cc:1035] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:18.447187 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.441655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:43141 (local address 127.7.157.67:33521)
0929 18:42:18.441764 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.441769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.441783 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:42:18.441822 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.441825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.441871 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.441935 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:18.444533 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.445251 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.446071 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.446360 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.446446 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.446733 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:18.446817 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:18.446881 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.446984 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.447039 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:42:18.447592 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.440589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:33521 (local address 127.7.157.65:43141)
0929 18:42:18.441084 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:18.441100 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:18.441197 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:18.444299 (+ 3102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:18.444302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:18.444312 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:42:18.444423 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.444429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.445346 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.445348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.445953 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.445960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.446457 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.446460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.446546 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.446591 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:42:18.446899 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:42:18.446905 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:42:18.446963 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:42:18.447447 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":98,"threads_started":1}
I20260929 18:42:18.475117 9538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:18.475312 9538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:18.477975 9538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1: Sending DeleteTablet for 3 replicas of tablet 6d1c452bd5a34b4a82704e170956c4a9
I20260929 18:42:18.478749 9955 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d1c452bd5a34b4a82704e170956c4a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:18 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:58262
I20260929 18:42:18.478924 9675 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d1c452bd5a34b4a82704e170956c4a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:18 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:57540
I20260929 18:42:18.479400 10091 tablet_replica.cc:330] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2: stopping tablet replica
I20260929 18:42:18.479775 10091 raft_consensus.cc:2229] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:42:18.479898 9815 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d1c452bd5a34b4a82704e170956c4a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:18 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:47536
I20260929 18:42:18.480288 10091 raft_consensus.cc:2258] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:18.481129 9538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:42:18.481585 9538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:18.482262 10091 ts_tablet_manager.cc:1905] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:18.485766 10091 ts_tablet_manager.cc:1918] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:42:18.485982 10093 tablet_replica.cc:330] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0: stopping tablet replica
I20260929 18:42:18.486353 10093 raft_consensus.cc:2229] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:18.486506 10091 log.cc:1196] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/6d1c452bd5a34b4a82704e170956c4a9
I20260929 18:42:18.486872 10091 ts_tablet_manager.cc:1939] T 6d1c452bd5a34b4a82704e170956c4a9 P 1c92df9ae320465d90fbf8ceb89a66c2: Deleting consensus metadata
I20260929 18:42:18.487181 10093 raft_consensus.cc:2258] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:18.487188 9955 tablet_service.cc:1467] Processing CreateTablet for tablet e5ce599c1b734471b58b261684ff6551 (DEFAULT_TABLE table=test-table [id=66fd06ac925e4cb7be12ca88903787cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:18.487475 9955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ce599c1b734471b58b261684ff6551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:18.487782 9675 tablet_service.cc:1467] Processing CreateTablet for tablet e5ce599c1b734471b58b261684ff6551 (DEFAULT_TABLE table=test-table [id=66fd06ac925e4cb7be12ca88903787cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:18.488046 9675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ce599c1b734471b58b261684ff6551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:18.489620 10045 tablet_bootstrap.cc:492] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2: Bootstrap starting.
I20260929 18:42:18.490527 10045 tablet_bootstrap.cc:654] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:18.491117 10044 tablet_bootstrap.cc:492] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc: Bootstrap starting.
I20260929 18:42:18.492089 10044 tablet_bootstrap.cc:654] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:18.492956 9522 catalog_manager.cc:4841] TS 1c92df9ae320465d90fbf8ceb89a66c2 (127.7.157.67:33521): tablet 6d1c452bd5a34b4a82704e170956c4a9 (table test-table [id=2c40585dcd2e4a42b3618aeea9fcf172]) successfully deleted
I20260929 18:42:18.490890 9815 tablet_service.cc:1467] Processing CreateTablet for tablet e5ce599c1b734471b58b261684ff6551 (DEFAULT_TABLE table=test-table [id=66fd06ac925e4cb7be12ca88903787cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:18.493395 9815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ce599c1b734471b58b261684ff6551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:18.494130 10093 ts_tablet_manager.cc:1905] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:18.495739 10046 tablet_bootstrap.cc:492] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0: Bootstrap starting.
I20260929 18:42:18.496565 10046 tablet_bootstrap.cc:654] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:18.497341 10045 tablet_bootstrap.cc:492] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2: No bootstrap required, opened a new log
I20260929 18:42:18.497424 10045 ts_tablet_manager.cc:1397] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260929 18:42:18.497862 10045 raft_consensus.cc:348] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.497967 10045 raft_consensus.cc:374] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:18.497996 10045 raft_consensus.cc:729] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c92df9ae320465d90fbf8ceb89a66c2, State: Initialized, Role: FOLLOWER
I20260929 18:42:18.498111 10045 consensus_queue.cc:260] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.498350 10045 ts_tablet_manager.cc:1428] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:18.499413 10044 tablet_bootstrap.cc:492] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc: No bootstrap required, opened a new log
I20260929 18:42:18.499496 10044 ts_tablet_manager.cc:1397] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260929 18:42:18.499905 10044 raft_consensus.cc:348] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.500008 10044 raft_consensus.cc:374] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:18.500059 10044 raft_consensus.cc:729] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef92c246c8b34f49a93d34818aa8eebc, State: Initialized, Role: FOLLOWER
I20260929 18:42:18.500195 10044 consensus_queue.cc:260] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.500445 10044 ts_tablet_manager.cc:1428] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:42:18.501269 10093 ts_tablet_manager.cc:1918] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:42:18.501428 10093 log.cc:1196] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/6d1c452bd5a34b4a82704e170956c4a9
I20260929 18:42:18.501812 10093 ts_tablet_manager.cc:1939] T 6d1c452bd5a34b4a82704e170956c4a9 P 388b72481aa04578b149bc0f93ec33f0: Deleting consensus metadata
I20260929 18:42:18.503270 9525 catalog_manager.cc:4841] TS 388b72481aa04578b149bc0f93ec33f0 (127.7.157.66:38575): tablet 6d1c452bd5a34b4a82704e170956c4a9 (table test-table [id=2c40585dcd2e4a42b3618aeea9fcf172]) successfully deleted
I20260929 18:42:18.503598 10046 tablet_bootstrap.cc:492] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0: No bootstrap required, opened a new log
I20260929 18:42:18.503680 10046 ts_tablet_manager.cc:1397] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260929 18:42:18.504108 10046 raft_consensus.cc:348] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.504220 10046 raft_consensus.cc:374] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:18.504247 10046 raft_consensus.cc:729] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 388b72481aa04578b149bc0f93ec33f0, State: Initialized, Role: FOLLOWER
I20260929 18:42:18.504357 10046 consensus_queue.cc:260] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.504590 10046 ts_tablet_manager.cc:1428] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:18.505312 10092 tablet_replica.cc:330] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc: stopping tablet replica
I20260929 18:42:18.505645 10092 raft_consensus.cc:2229] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:18.505793 10092 raft_consensus.cc:2258] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:18.507771 10092 ts_tablet_manager.cc:1905] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:18.510706 10092 ts_tablet_manager.cc:1918] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:42:18.510807 10092 log.cc:1196] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/wal/wals/6d1c452bd5a34b4a82704e170956c4a9
I20260929 18:42:18.511143 10092 ts_tablet_manager.cc:1939] T 6d1c452bd5a34b4a82704e170956c4a9 P ef92c246c8b34f49a93d34818aa8eebc: Deleting consensus metadata
I20260929 18:42:18.512458 9524 catalog_manager.cc:4841] TS ef92c246c8b34f49a93d34818aa8eebc (127.7.157.65:39735): tablet 6d1c452bd5a34b4a82704e170956c4a9 (table test-table [id=2c40585dcd2e4a42b3618aeea9fcf172]) successfully deleted
I20260929 18:42:18.606235 10078 raft_consensus.cc:482] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:18.606397 10078 raft_consensus.cc:504] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.606846 10078 leader_election.cc:288] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 388b72481aa04578b149bc0f93ec33f0 (127.7.157.66:38575), 1c92df9ae320465d90fbf8ceb89a66c2 (127.7.157.67:33521)
I20260929 18:42:18.607295 9975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ce599c1b734471b58b261684ff6551" candidate_uuid: "ef92c246c8b34f49a93d34818aa8eebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" is_pre_election: true
I20260929 18:42:18.607300 9835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ce599c1b734471b58b261684ff6551" candidate_uuid: "ef92c246c8b34f49a93d34818aa8eebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "388b72481aa04578b149bc0f93ec33f0" is_pre_election: true
I20260929 18:42:18.607438 9835 raft_consensus.cc:2454] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef92c246c8b34f49a93d34818aa8eebc in term 0.
I20260929 18:42:18.607437 9975 raft_consensus.cc:2454] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef92c246c8b34f49a93d34818aa8eebc in term 0.
I20260929 18:42:18.607735 9626 leader_election.cc:302] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c92df9ae320465d90fbf8ceb89a66c2, ef92c246c8b34f49a93d34818aa8eebc; no voters:
I20260929 18:42:18.607918 10078 raft_consensus.cc:2781] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:18.607969 10078 raft_consensus.cc:482] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:18.607996 10078 raft_consensus.cc:3037] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:18.608965 10078 raft_consensus.cc:504] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.609289 10078 leader_election.cc:288] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [CANDIDATE]: Term 1 election: Requested vote from peers 388b72481aa04578b149bc0f93ec33f0 (127.7.157.66:38575), 1c92df9ae320465d90fbf8ceb89a66c2 (127.7.157.67:33521)
I20260929 18:42:18.609639 9975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ce599c1b734471b58b261684ff6551" candidate_uuid: "ef92c246c8b34f49a93d34818aa8eebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c92df9ae320465d90fbf8ceb89a66c2"
I20260929 18:42:18.609650 9835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ce599c1b734471b58b261684ff6551" candidate_uuid: "ef92c246c8b34f49a93d34818aa8eebc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "388b72481aa04578b149bc0f93ec33f0"
I20260929 18:42:18.609750 9975 raft_consensus.cc:3037] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:18.609791 9835 raft_consensus.cc:3037] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:18.610729 9975 raft_consensus.cc:2454] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef92c246c8b34f49a93d34818aa8eebc in term 1.
I20260929 18:42:18.610985 9835 raft_consensus.cc:2454] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef92c246c8b34f49a93d34818aa8eebc in term 1.
I20260929 18:42:18.610997 9626 leader_election.cc:302] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c92df9ae320465d90fbf8ceb89a66c2, ef92c246c8b34f49a93d34818aa8eebc; no voters:
I20260929 18:42:18.611177 10078 raft_consensus.cc:2781] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:18.611253 10078 raft_consensus.cc:686] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [term 1 LEADER]: Becoming Leader. State: Replica: ef92c246c8b34f49a93d34818aa8eebc, State: Running, Role: LEADER
I20260929 18:42:18.611389 10078 consensus_queue.cc:237] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:18.613112 9538 catalog_manager.cc:5495] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc reported cstate change: term changed from 0 to 1, leader changed from <none> to ef92c246c8b34f49a93d34818aa8eebc (127.7.157.65). New cstate: current_term: 1 leader_uuid: "ef92c246c8b34f49a93d34818aa8eebc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } health_report { overall_health: UNKNOWN } } }
I20260929 18:42:18.636041 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.633102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.7.157.65:39735)
0929 18:42:18.633203 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.633207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.633238 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:42:18.633371 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.633374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.633420 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.633479 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:18.633703 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.634206 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.635012 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.635290 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.635376 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.635618 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:18.635689 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:18.635744 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.635842 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.635893 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:42:18.644011 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.641332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:39735 (local address 127.7.157.65:58501)
0929 18:42:18.641464 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:18.641477 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:18.641538 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:18.641721 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:18.641723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:18.641727 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:42:18.641800 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.641804 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.642424 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.642426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.643042 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:18.643050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.643459 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.643462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:18.643524 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.643544 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:42:18.643756 (+ 212us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:42:18.643761 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:42:18.643817 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:42:18.643894 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 18:42:18.644011 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:18.641394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:58501 (local address 127.7.157.65:39735)
0929 18:42:18.641492 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:18.641497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:18.641509 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:42:18.641612 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:18.641615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:18.641652 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:18.641711 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:18.641865 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.642355 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.643168 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:18.643365 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:18.643443 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:18.643632 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:18.643693 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:18.643748 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:18.643823 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 18:42:18.643869 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":33}
I20260929 18:42:18.649678 9835 raft_consensus.cc:1261] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0 [term 1 FOLLOWER]: Refusing update from remote peer ef92c246c8b34f49a93d34818aa8eebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:18.650092 10051 consensus_queue.cc:1035] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:18.650426 9975 raft_consensus.cc:1261] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Refusing update from remote peer ef92c246c8b34f49a93d34818aa8eebc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:18.650791 10051 consensus_queue.cc:1035] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:42:18.663419 10072 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 18:42:20.216832 9537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:20.216989 9537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:20.219211 9537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1: Sending DeleteTablet for 3 replicas of tablet e5ce599c1b734471b58b261684ff6551
I20260929 18:42:20.219714 9675 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ce599c1b734471b58b261684ff6551 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:20 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:57540
I20260929 18:42:20.219833 9815 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ce599c1b734471b58b261684ff6551 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:20 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:47536
I20260929 18:42:20.219930 9955 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ce599c1b734471b58b261684ff6551 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:20 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:58262
I20260929 18:42:20.220185 10123 tablet_replica.cc:330] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc: stopping tablet replica
I20260929 18:42:20.220309 10123 raft_consensus.cc:2229] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:42:20.220687 10123 raft_consensus.cc:2258] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:20.220731 10125 tablet_replica.cc:330] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2: stopping tablet replica
I20260929 18:42:20.220861 10125 raft_consensus.cc:2229] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:20.220996 10125 raft_consensus.cc:2258] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:20.221369 10123 ts_tablet_manager.cc:1905] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:20.221609 10125 ts_tablet_manager.cc:1905] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:20.223250 10124 tablet_replica.cc:330] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0: stopping tablet replica
I20260929 18:42:20.223413 10124 raft_consensus.cc:2229] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:20.223608 10124 raft_consensus.cc:2258] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:20.224046 10123 ts_tablet_manager.cc:1918] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:42:20.224150 10123 log.cc:1196] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/wal/wals/e5ce599c1b734471b58b261684ff6551
I20260929 18:42:20.224226 10125 ts_tablet_manager.cc:1918] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:42:20.224326 10125 log.cc:1196] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/e5ce599c1b734471b58b261684ff6551
I20260929 18:42:20.224574 10125 ts_tablet_manager.cc:1939] T e5ce599c1b734471b58b261684ff6551 P 1c92df9ae320465d90fbf8ceb89a66c2: Deleting consensus metadata
I20260929 18:42:20.224977 10124 ts_tablet_manager.cc:1905] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:20.225524 9522 catalog_manager.cc:4841] TS 1c92df9ae320465d90fbf8ceb89a66c2 (127.7.157.67:33521): tablet e5ce599c1b734471b58b261684ff6551 (table test-table [id=66fd06ac925e4cb7be12ca88903787cd]) successfully deleted
I20260929 18:42:20.225834 10123 ts_tablet_manager.cc:1939] T e5ce599c1b734471b58b261684ff6551 P ef92c246c8b34f49a93d34818aa8eebc: Deleting consensus metadata
I20260929 18:42:20.226542 9524 catalog_manager.cc:4841] TS ef92c246c8b34f49a93d34818aa8eebc (127.7.157.65:39735): tablet e5ce599c1b734471b58b261684ff6551 (table test-table [id=66fd06ac925e4cb7be12ca88903787cd]) successfully deleted
I20260929 18:42:20.227483 10124 ts_tablet_manager.cc:1918] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:42:20.227556 10124 log.cc:1196] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/e5ce599c1b734471b58b261684ff6551
I20260929 18:42:20.227775 10124 ts_tablet_manager.cc:1939] T e5ce599c1b734471b58b261684ff6551 P 388b72481aa04578b149bc0f93ec33f0: Deleting consensus metadata
I20260929 18:42:20.228447 9525 catalog_manager.cc:4841] TS 388b72481aa04578b149bc0f93ec33f0 (127.7.157.66:38575): tablet e5ce599c1b734471b58b261684ff6551 (table test-table [id=66fd06ac925e4cb7be12ca88903787cd]) successfully deleted
Sep 29 18:42:20 dist-test-slave-smpn krb5kdc[9493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707340, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:42:20 dist-test-slave-smpn krb5kdc[9493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707340, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:20.252506 10133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:20.239257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.7.157.126:37709)
0929 18:42:20.239498 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:20.239505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:20.239540 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:42:20.239686 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:20.239688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:20.239733 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:20.239813 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:20.240453 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.240986 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.241773 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.241984 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.242053 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:20.249582 (+ 7529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:20.249586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:20.249589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:20.249637 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:20.251037 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:20.251503 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:20.251506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:20.251507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:20.251585 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:20.251805 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:20.251807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:20.251808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:20.251986 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:20.252083 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:20.252266 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:42:20.252370 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
I20260929 18:42:20.255282 9537 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:42:20.255767 9537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:20.260103 9675 tablet_service.cc:1467] Processing CreateTablet for tablet 05194911c5344f678f84cff05ac4950c (DEFAULT_TABLE table=test-table [id=f804af2da7da4de8bc9282bd4a268a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:20.260212 9815 tablet_service.cc:1467] Processing CreateTablet for tablet 05194911c5344f678f84cff05ac4950c (DEFAULT_TABLE table=test-table [id=f804af2da7da4de8bc9282bd4a268a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:20.260432 9675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05194911c5344f678f84cff05ac4950c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:20.260485 9815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05194911c5344f678f84cff05ac4950c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:20.260485 9955 tablet_service.cc:1467] Processing CreateTablet for tablet 05194911c5344f678f84cff05ac4950c (DEFAULT_TABLE table=test-table [id=f804af2da7da4de8bc9282bd4a268a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:20.260730 9955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05194911c5344f678f84cff05ac4950c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:20.262523 10134 tablet_bootstrap.cc:492] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0: Bootstrap starting.
I20260929 18:42:20.262996 10135 tablet_bootstrap.cc:492] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc: Bootstrap starting.
I20260929 18:42:20.263003 10136 tablet_bootstrap.cc:492] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2: Bootstrap starting.
I20260929 18:42:20.263640 10134 tablet_bootstrap.cc:654] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:20.263772 10136 tablet_bootstrap.cc:654] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:20.263908 10135 tablet_bootstrap.cc:654] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:20.264947 10136 tablet_bootstrap.cc:492] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2: No bootstrap required, opened a new log
I20260929 18:42:20.264945 10134 tablet_bootstrap.cc:492] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0: No bootstrap required, opened a new log
I20260929 18:42:20.265033 10134 ts_tablet_manager.cc:1397] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:42:20.265033 10136 ts_tablet_manager.cc:1397] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:20.265220 10135 tablet_bootstrap.cc:492] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc: No bootstrap required, opened a new log
I20260929 18:42:20.265301 10135 ts_tablet_manager.cc:1397] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:20.265442 10136 raft_consensus.cc:348] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.265496 10134 raft_consensus.cc:348] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.265574 10136 raft_consensus.cc:374] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:20.265596 10134 raft_consensus.cc:374] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:20.265609 10136 raft_consensus.cc:729] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c92df9ae320465d90fbf8ceb89a66c2, State: Initialized, Role: FOLLOWER
I20260929 18:42:20.265627 10134 raft_consensus.cc:729] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 388b72481aa04578b149bc0f93ec33f0, State: Initialized, Role: FOLLOWER
I20260929 18:42:20.265686 10135 raft_consensus.cc:348] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.265726 10136 consensus_queue.cc:260] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.265776 10135 raft_consensus.cc:374] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:20.265806 10135 raft_consensus.cc:729] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef92c246c8b34f49a93d34818aa8eebc, State: Initialized, Role: FOLLOWER
I20260929 18:42:20.265776 10134 consensus_queue.cc:260] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.265916 10135 consensus_queue.cc:260] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.266086 10136 ts_tablet_manager.cc:1428] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:20.266151 10134 ts_tablet_manager.cc:1428] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:20.266149 10135 ts_tablet_manager.cc:1428] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:42:20.429330 10141 raft_consensus.cc:482] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:20.429481 10141 raft_consensus.cc:504] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.431159 10141 leader_election.cc:288] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef92c246c8b34f49a93d34818aa8eebc (127.7.157.65:39735), 1c92df9ae320465d90fbf8ceb89a66c2 (127.7.157.67:33521)
I20260929 18:42:20.435120 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:20.430835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:33521 (local address 127.7.157.66:59691)
0929 18:42:20.431259 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:20.431275 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:20.431387 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:20.431681 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:20.431684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:20.431730 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:20.431863 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:20.431868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.432769 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.432773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:20.433927 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:20.433938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.434713 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.434716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:20.434787 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:20.434808 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:42:20.434863 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 18:42:20.434923 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"spinlock_wait_cycles":6528,"thread_start_us":142,"threads_started":1}
I20260929 18:42:20.435137 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:20.431015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:59691 (local address 127.7.157.67:33521)
0929 18:42:20.431387 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:20.431396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:20.431412 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:42:20.431468 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:20.431471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:20.431516 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:20.431962 (+ 446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:20.432066 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.432659 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.434063 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.434590 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.434720 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:20.434774 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:20.434894 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 18:42:20.434971 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":148,"threads_started":1}
I20260929 18:42:20.435482 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:20.430838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:39735 (local address 127.7.157.66:45233)
0929 18:42:20.431613 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:20.431626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:20.431726 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:20.431998 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:20.432000 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:20.432016 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:20.432119 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:20.432125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.433182 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.433186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:20.434219 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:20.434226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.435045 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.435048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:20.435185 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:20.435208 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:42:20.435276 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:42:20.435343 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":34,"spinlock_wait_cycles":29824,"thread_start_us":98,"threads_started":1}
I20260929 18:42:20.435601 9974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05194911c5344f678f84cff05ac4950c" candidate_uuid: "388b72481aa04578b149bc0f93ec33f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" is_pre_election: true
I20260929 18:42:20.435767 9974 raft_consensus.cc:2454] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 388b72481aa04578b149bc0f93ec33f0 in term 0.
I20260929 18:42:20.435809 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:20.430922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:45233 (local address 127.7.157.65:39735)
0929 18:42:20.431520 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:20.431530 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:20.431693 (+ 163us) server_negotiation.cc:407] Connection header received
0929 18:42:20.431811 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:20.431814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:20.431869 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:20.431966 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:20.432221 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.433028 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.434334 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.434910 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.435022 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:20.435090 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:20.435618 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 18:42:20.435700 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":152,"threads_started":1}
I20260929 18:42:20.436146 9766 leader_election.cc:302] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c92df9ae320465d90fbf8ceb89a66c2, 388b72481aa04578b149bc0f93ec33f0; no voters:
I20260929 18:42:20.436156 9695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05194911c5344f678f84cff05ac4950c" candidate_uuid: "388b72481aa04578b149bc0f93ec33f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef92c246c8b34f49a93d34818aa8eebc" is_pre_election: true
I20260929 18:42:20.436277 9695 raft_consensus.cc:2454] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 388b72481aa04578b149bc0f93ec33f0 in term 0.
I20260929 18:42:20.436385 10141 raft_consensus.cc:2781] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:20.436439 10141 raft_consensus.cc:482] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:20.436474 10141 raft_consensus.cc:3037] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:20.437498 10141 raft_consensus.cc:504] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.437865 10141 leader_election.cc:288] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [CANDIDATE]: Term 1 election: Requested vote from peers ef92c246c8b34f49a93d34818aa8eebc (127.7.157.65:39735), 1c92df9ae320465d90fbf8ceb89a66c2 (127.7.157.67:33521)
I20260929 18:42:20.438149 9695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05194911c5344f678f84cff05ac4950c" candidate_uuid: "388b72481aa04578b149bc0f93ec33f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef92c246c8b34f49a93d34818aa8eebc"
I20260929 18:42:20.438259 9695 raft_consensus.cc:3037] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:20.438305 9974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05194911c5344f678f84cff05ac4950c" candidate_uuid: "388b72481aa04578b149bc0f93ec33f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c92df9ae320465d90fbf8ceb89a66c2"
I20260929 18:42:20.438426 9974 raft_consensus.cc:3037] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:20.439214 9695 raft_consensus.cc:2454] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 388b72481aa04578b149bc0f93ec33f0 in term 1.
I20260929 18:42:20.439462 9974 raft_consensus.cc:2454] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 388b72481aa04578b149bc0f93ec33f0 in term 1.
I20260929 18:42:20.439471 9768 leader_election.cc:302] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 388b72481aa04578b149bc0f93ec33f0, ef92c246c8b34f49a93d34818aa8eebc; no voters:
I20260929 18:42:20.439729 10141 raft_consensus.cc:2781] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:20.439914 10141 raft_consensus.cc:686] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [term 1 LEADER]: Becoming Leader. State: Replica: 388b72481aa04578b149bc0f93ec33f0, State: Running, Role: LEADER
I20260929 18:42:20.440145 10141 consensus_queue.cc:237] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.442544 9537 catalog_manager.cc:5495] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 388b72481aa04578b149bc0f93ec33f0 (127.7.157.66). New cstate: current_term: 1 leader_uuid: "388b72481aa04578b149bc0f93ec33f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } health_report { overall_health: UNKNOWN } } }
I20260929 18:42:20.464983 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:20.461478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47544 (local address 127.7.157.66:38575)
0929 18:42:20.461718 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:20.461724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:20.461738 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:42:20.461804 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:20.461806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:20.461856 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:20.461956 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:20.462179 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.462725 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.463535 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.463806 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.463887 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:20.464076 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:20.464608 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:20.464694 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:20.464772 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 18:42:20.464823 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":101,"threads_started":1}
I20260929 18:42:20.474723 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:20.472003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:59851 (local address 127.7.157.65:39735)
0929 18:42:20.472119 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:20.472123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:20.472152 (+ 29us) server_negotiation.cc:407] Connection header received
0929 18:42:20.472256 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:20.472258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:20.472302 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:20.472355 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:20.472541 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.473044 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.473858 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.474066 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.474138 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:20.474364 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:20.474431 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:20.474476 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:20.474553 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 18:42:20.474613 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:42:20.474736 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:20.471804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:39735 (local address 127.7.157.66:59851)
0929 18:42:20.472075 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:20.472087 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:20.472175 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:20.472365 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:20.472367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:20.472372 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:42:20.472473 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:20.472477 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.473127 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.473129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:20.473758 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:20.473766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.474157 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.474159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:20.474226 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:20.474268 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:42:20.474484 (+ 216us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:42:20.474497 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 18:42:20.474540 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 18:42:20.474603 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
I20260929 18:42:20.480664 9695 raft_consensus.cc:1261] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc [term 1 FOLLOWER]: Refusing update from remote peer 388b72481aa04578b149bc0f93ec33f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:20.480756 9974 raft_consensus.cc:1261] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Refusing update from remote peer 388b72481aa04578b149bc0f93ec33f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:20.481392 10146 consensus_queue.cc:1035] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:20.481577 10141 consensus_queue.cc:1035] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:20.497782 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:20.494668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:51291 (local address 127.7.157.66:38575)
0929 18:42:20.494787 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:20.494791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:20.494967 (+ 176us) server_negotiation.cc:407] Connection header received
0929 18:42:20.495094 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:20.495097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:20.495146 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:20.495202 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:20.495526 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.496029 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.496839 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.497066 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.497150 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:20.497382 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:20.497455 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:20.497511 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:20.497594 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 18:42:20.497644 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:42:20.497794 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:20.494606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:38575 (local address 127.7.157.65:51291)
0929 18:42:20.494885 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:20.494897 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:20.494991 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:20.495321 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:20.495324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:20.495332 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:42:20.495432 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:20.495437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.496127 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.496131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:20.496736 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:20.496745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:20.497157 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:20.497161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:20.497254 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:20.497280 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:42:20.497522 (+ 242us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:42:20.497528 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:42:20.497586 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:42:20.497670 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":98,"threads_started":1}
I20260929 18:42:20.520257 9538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:20.520404 9538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:20.522435 9538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1: Sending DeleteTablet for 3 replicas of tablet 05194911c5344f678f84cff05ac4950c
I20260929 18:42:20.522971 9675 tablet_service.cc:1514] Processing DeleteTablet for tablet 05194911c5344f678f84cff05ac4950c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:20 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:57540
I20260929 18:42:20.523094 9955 tablet_service.cc:1514] Processing DeleteTablet for tablet 05194911c5344f678f84cff05ac4950c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:20 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:58262
I20260929 18:42:20.523203 10123 tablet_replica.cc:330] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc: stopping tablet replica
I20260929 18:42:20.523319 10123 raft_consensus.cc:2229] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:20.523430 10123 raft_consensus.cc:2258] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:20.523535 10125 tablet_replica.cc:330] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2: stopping tablet replica
I20260929 18:42:20.523650 10125 raft_consensus.cc:2229] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:20.523764 10125 raft_consensus.cc:2258] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:20.523898 9815 tablet_service.cc:1514] Processing DeleteTablet for tablet 05194911c5344f678f84cff05ac4950c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:20 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:47536
I20260929 18:42:20.524068 10124 tablet_replica.cc:330] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0: stopping tablet replica
I20260929 18:42:20.524101 9538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:42:20.524183 10124 raft_consensus.cc:2229] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [term 1 LEADER]: Raft consensus shutting down.
W20260929 18:42:20.524437 9538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:20.524470 10124 raft_consensus.cc:2258] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:20.525362 10123 ts_tablet_manager.cc:1905] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:20.525507 10125 ts_tablet_manager.cc:1905] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:20.527664 10125 ts_tablet_manager.cc:1918] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:42:20.527715 10125 log.cc:1196] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/05194911c5344f678f84cff05ac4950c
I20260929 18:42:20.527889 10125 ts_tablet_manager.cc:1939] T 05194911c5344f678f84cff05ac4950c P 1c92df9ae320465d90fbf8ceb89a66c2: Deleting consensus metadata
I20260929 18:42:20.528625 10123 ts_tablet_manager.cc:1918] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:42:20.528692 10123 log.cc:1196] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-0/wal/wals/05194911c5344f678f84cff05ac4950c
I20260929 18:42:20.528692 9522 catalog_manager.cc:4841] TS 1c92df9ae320465d90fbf8ceb89a66c2 (127.7.157.67:33521): tablet 05194911c5344f678f84cff05ac4950c (table test-table [id=f804af2da7da4de8bc9282bd4a268a91]) successfully deleted
I20260929 18:42:20.529790 10124 ts_tablet_manager.cc:1905] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:20.532212 10124 ts_tablet_manager.cc:1918] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:42:20.532280 10124 log.cc:1196] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/05194911c5344f678f84cff05ac4950c
I20260929 18:42:20.532502 10124 ts_tablet_manager.cc:1939] T 05194911c5344f678f84cff05ac4950c P 388b72481aa04578b149bc0f93ec33f0: Deleting consensus metadata
I20260929 18:42:20.532630 9675 tablet_service.cc:1467] Processing CreateTablet for tablet 4597de11aa854ca297420474e7f7fb21 (DEFAULT_TABLE table=test-table [id=9323bc48399e40d4b60db3bd1befbf2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:20.532793 9955 tablet_service.cc:1467] Processing CreateTablet for tablet 4597de11aa854ca297420474e7f7fb21 (DEFAULT_TABLE table=test-table [id=9323bc48399e40d4b60db3bd1befbf2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:20.532891 9675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4597de11aa854ca297420474e7f7fb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:20.533025 9955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4597de11aa854ca297420474e7f7fb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:20.533783 10123 ts_tablet_manager.cc:1939] T 05194911c5344f678f84cff05ac4950c P ef92c246c8b34f49a93d34818aa8eebc: Deleting consensus metadata
I20260929 18:42:20.535142 10135 tablet_bootstrap.cc:492] T 4597de11aa854ca297420474e7f7fb21 P ef92c246c8b34f49a93d34818aa8eebc: Bootstrap starting.
I20260929 18:42:20.535226 9524 catalog_manager.cc:4841] TS ef92c246c8b34f49a93d34818aa8eebc (127.7.157.65:39735): tablet 05194911c5344f678f84cff05ac4950c (table test-table [id=f804af2da7da4de8bc9282bd4a268a91]) successfully deleted
I20260929 18:42:20.535416 9525 catalog_manager.cc:4841] TS 388b72481aa04578b149bc0f93ec33f0 (127.7.157.66:38575): tablet 05194911c5344f678f84cff05ac4950c (table test-table [id=f804af2da7da4de8bc9282bd4a268a91]) successfully deleted
I20260929 18:42:20.535854 9815 tablet_service.cc:1467] Processing CreateTablet for tablet 4597de11aa854ca297420474e7f7fb21 (DEFAULT_TABLE table=test-table [id=9323bc48399e40d4b60db3bd1befbf2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:20.535941 10135 tablet_bootstrap.cc:654] T 4597de11aa854ca297420474e7f7fb21 P ef92c246c8b34f49a93d34818aa8eebc: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:20.536089 9815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4597de11aa854ca297420474e7f7fb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:20.536505 10136 tablet_bootstrap.cc:492] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2: Bootstrap starting.
I20260929 18:42:20.537303 10136 tablet_bootstrap.cc:654] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:20.538087 10134 tablet_bootstrap.cc:492] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0: Bootstrap starting.
I20260929 18:42:20.538308 10135 tablet_bootstrap.cc:492] T 4597de11aa854ca297420474e7f7fb21 P ef92c246c8b34f49a93d34818aa8eebc: No bootstrap required, opened a new log
I20260929 18:42:20.538383 10135 ts_tablet_manager.cc:1397] T 4597de11aa854ca297420474e7f7fb21 P ef92c246c8b34f49a93d34818aa8eebc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:42:20.538774 10135 raft_consensus.cc:348] T 4597de11aa854ca297420474e7f7fb21 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.538878 10135 raft_consensus.cc:374] T 4597de11aa854ca297420474e7f7fb21 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:20.538908 10135 raft_consensus.cc:729] T 4597de11aa854ca297420474e7f7fb21 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef92c246c8b34f49a93d34818aa8eebc, State: Initialized, Role: FOLLOWER
I20260929 18:42:20.538942 10134 tablet_bootstrap.cc:654] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:20.539027 10135 consensus_queue.cc:260] T 4597de11aa854ca297420474e7f7fb21 P ef92c246c8b34f49a93d34818aa8eebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.539311 10135 ts_tablet_manager.cc:1428] T 4597de11aa854ca297420474e7f7fb21 P ef92c246c8b34f49a93d34818aa8eebc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:20.540105 10134 tablet_bootstrap.cc:492] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0: No bootstrap required, opened a new log
I20260929 18:42:20.540130 10136 tablet_bootstrap.cc:492] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2: No bootstrap required, opened a new log
I20260929 18:42:20.540167 10134 ts_tablet_manager.cc:1397] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:20.540207 10136 ts_tablet_manager.cc:1397] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 18:42:20.540474 10134 raft_consensus.cc:348] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.540544 10134 raft_consensus.cc:374] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:20.540586 10134 raft_consensus.cc:729] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 388b72481aa04578b149bc0f93ec33f0, State: Initialized, Role: FOLLOWER
I20260929 18:42:20.540563 10136 raft_consensus.cc:348] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.540661 10136 raft_consensus.cc:374] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:20.540690 10136 raft_consensus.cc:729] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c92df9ae320465d90fbf8ceb89a66c2, State: Initialized, Role: FOLLOWER
I20260929 18:42:20.540714 10134 consensus_queue.cc:260] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.540800 10136 consensus_queue.cc:260] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.541007 10134 ts_tablet_manager.cc:1428] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:20.541036 10136 ts_tablet_manager.cc:1428] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:20.624599 10141 raft_consensus.cc:482] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:20.624727 10141 raft_consensus.cc:504] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.625174 10141 leader_election.cc:288] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef92c246c8b34f49a93d34818aa8eebc (127.7.157.65:39735), 1c92df9ae320465d90fbf8ceb89a66c2 (127.7.157.67:33521)
I20260929 18:42:20.625622 9974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4597de11aa854ca297420474e7f7fb21" candidate_uuid: "388b72481aa04578b149bc0f93ec33f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" is_pre_election: true
I20260929 18:42:20.625628 9695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4597de11aa854ca297420474e7f7fb21" candidate_uuid: "388b72481aa04578b149bc0f93ec33f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef92c246c8b34f49a93d34818aa8eebc" is_pre_election: true
I20260929 18:42:20.625774 9974 raft_consensus.cc:2454] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 388b72481aa04578b149bc0f93ec33f0 in term 0.
I20260929 18:42:20.625774 9695 raft_consensus.cc:2454] T 4597de11aa854ca297420474e7f7fb21 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 388b72481aa04578b149bc0f93ec33f0 in term 0.
I20260929 18:42:20.626077 9768 leader_election.cc:302] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 388b72481aa04578b149bc0f93ec33f0, ef92c246c8b34f49a93d34818aa8eebc; no voters:
I20260929 18:42:20.626292 10141 raft_consensus.cc:2781] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:20.626351 10141 raft_consensus.cc:482] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:20.626389 10141 raft_consensus.cc:3037] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:20.627692 10141 raft_consensus.cc:504] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.628024 10141 leader_election.cc:288] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [CANDIDATE]: Term 1 election: Requested vote from peers ef92c246c8b34f49a93d34818aa8eebc (127.7.157.65:39735), 1c92df9ae320465d90fbf8ceb89a66c2 (127.7.157.67:33521)
I20260929 18:42:20.628383 9974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4597de11aa854ca297420474e7f7fb21" candidate_uuid: "388b72481aa04578b149bc0f93ec33f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c92df9ae320465d90fbf8ceb89a66c2"
I20260929 18:42:20.628438 9695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4597de11aa854ca297420474e7f7fb21" candidate_uuid: "388b72481aa04578b149bc0f93ec33f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef92c246c8b34f49a93d34818aa8eebc"
I20260929 18:42:20.628515 9974 raft_consensus.cc:3037] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:20.628541 9695 raft_consensus.cc:3037] T 4597de11aa854ca297420474e7f7fb21 P ef92c246c8b34f49a93d34818aa8eebc [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:20.629617 9974 raft_consensus.cc:2454] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 388b72481aa04578b149bc0f93ec33f0 in term 1.
I20260929 18:42:20.629715 9695 raft_consensus.cc:2454] T 4597de11aa854ca297420474e7f7fb21 P ef92c246c8b34f49a93d34818aa8eebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 388b72481aa04578b149bc0f93ec33f0 in term 1.
I20260929 18:42:20.629926 9766 leader_election.cc:302] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c92df9ae320465d90fbf8ceb89a66c2, 388b72481aa04578b149bc0f93ec33f0; no voters:
I20260929 18:42:20.630131 10141 raft_consensus.cc:2781] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:20.630223 10141 raft_consensus.cc:686] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [term 1 LEADER]: Becoming Leader. State: Replica: 388b72481aa04578b149bc0f93ec33f0, State: Running, Role: LEADER
I20260929 18:42:20.630349 10141 consensus_queue.cc:237] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } }
I20260929 18:42:20.632124 9535 catalog_manager.cc:5495] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 388b72481aa04578b149bc0f93ec33f0 (127.7.157.66). New cstate: current_term: 1 leader_uuid: "388b72481aa04578b149bc0f93ec33f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "388b72481aa04578b149bc0f93ec33f0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 } health_report { overall_health: UNKNOWN } } }
I20260929 18:42:20.671254 9975 raft_consensus.cc:1261] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Refusing update from remote peer 388b72481aa04578b149bc0f93ec33f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:20.671327 9695 raft_consensus.cc:1261] T 4597de11aa854ca297420474e7f7fb21 P ef92c246c8b34f49a93d34818aa8eebc [term 1 FOLLOWER]: Refusing update from remote peer 388b72481aa04578b149bc0f93ec33f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:20.671716 10155 consensus_queue.cc:1035] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef92c246c8b34f49a93d34818aa8eebc" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:20.671875 10141 consensus_queue.cc:1035] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c92df9ae320465d90fbf8ceb89a66c2" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:42:20.686502 10105 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 18:42:22.239558 9537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:22.239748 9537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:22.242290 9537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0016b60f610a4161b1bc830fbb800db1: Sending DeleteTablet for 3 replicas of tablet 4597de11aa854ca297420474e7f7fb21
I20260929 18:42:22.242971 9675 tablet_service.cc:1514] Processing DeleteTablet for tablet 4597de11aa854ca297420474e7f7fb21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:22 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:57540
I20260929 18:42:22.242972 9815 tablet_service.cc:1514] Processing DeleteTablet for tablet 4597de11aa854ca297420474e7f7fb21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:22 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:47536
I20260929 18:42:22.243327 9955 tablet_service.cc:1514] Processing DeleteTablet for tablet 4597de11aa854ca297420474e7f7fb21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:22 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:58262
I20260929 18:42:22.243369 10188 tablet_replica.cc:330] T 4597de11aa854ca297420474e7f7fb21 P ef92c246c8b34f49a93d34818aa8eebc: stopping tablet replica
I20260929 18:42:22.243374 10189 tablet_replica.cc:330] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0: stopping tablet replica
I20260929 18:42:22.243491 10188 raft_consensus.cc:2229] T 4597de11aa854ca297420474e7f7fb21 P ef92c246c8b34f49a93d34818aa8eebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:22.243512 10189 raft_consensus.cc:2229] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:42:22.243563 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 9610
I20260929 18:42:22.244259 10189 raft_consensus.cc:2258] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:22.244382 10190 tablet_replica.cc:330] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2: stopping tablet replica
I20260929 18:42:22.244491 10190 raft_consensus.cc:2229] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:22.244592 10190 raft_consensus.cc:2258] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:22.245023 10189 ts_tablet_manager.cc:1905] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:22.245184 10190 ts_tablet_manager.cc:1905] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:22.247697 10189 ts_tablet_manager.cc:1918] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:42:22.247696 10190 ts_tablet_manager.cc:1918] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:42:22.247778 10190 log.cc:1196] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/4597de11aa854ca297420474e7f7fb21
I20260929 18:42:22.247778 10189 log.cc:1196] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/4597de11aa854ca297420474e7f7fb21
I20260929 18:42:22.248023 10189 ts_tablet_manager.cc:1939] T 4597de11aa854ca297420474e7f7fb21 P 388b72481aa04578b149bc0f93ec33f0: Deleting consensus metadata
I20260929 18:42:22.248015 10190 ts_tablet_manager.cc:1939] T 4597de11aa854ca297420474e7f7fb21 P 1c92df9ae320465d90fbf8ceb89a66c2: Deleting consensus metadata
I20260929 18:42:22.248867 9522 catalog_manager.cc:4841] TS 1c92df9ae320465d90fbf8ceb89a66c2 (127.7.157.67:33521): tablet 4597de11aa854ca297420474e7f7fb21 (table test-table [id=9323bc48399e40d4b60db3bd1befbf2b]) successfully deleted
I20260929 18:42:22.248924 9525 catalog_manager.cc:4841] TS 388b72481aa04578b149bc0f93ec33f0 (127.7.157.66:38575): tablet 4597de11aa854ca297420474e7f7fb21 (table test-table [id=9323bc48399e40d4b60db3bd1befbf2b]) successfully deleted
W20260929 18:42:22.250703 9524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:39735 (error 108)
I20260929 18:42:22.251400 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 9750
I20260929 18:42:22.252358 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:22.251418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:39735 (local address 127.0.0.1:57564)
0929 18:42:22.252179 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:22.252257 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:39735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":82,"threads_started":1}
W20260929 18:42:22.252570 9524 catalog_manager.cc:4584] TS ef92c246c8b34f49a93d34818aa8eebc (127.7.157.65:39735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4597de11aa854ca297420474e7f7fb21: Network error: Client connection negotiation failed: client connection to 127.7.157.65:39735: connect: Connection refused (error 111)
I20260929 18:42:22.257567 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 9890
I20260929 18:42:22.263221 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 9506
2026-09-29T18:42:22Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6060 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:42:22 dist-test-slave-smpn krb5kdc[10195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:42:22 dist-test-slave-smpn krb5kdc[10195](info): set up 2 sockets
Sep 29 18:42:22 dist-test-slave-smpn krb5kdc[10195](info): commencing operation
krb5kdc: starting...
W20260929 18:42:23.230737 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:42:23 dist-test-slave-smpn krb5kdc[10195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:42:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:42:23.346300 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:34891
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:34891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:23.453375 10208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:23.453632 10208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:23.453683 10208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:23.457599 10208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:23.457659 10208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:23.457684 10208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:23.457703 10208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:23.457723 10208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:23.462858 10208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:34891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:34891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:23.463886 10208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:23.464874 10208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:23.472970 10213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:23.473032 10214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:23.472972 10216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:23.473163 10208 server_base.cc:1034] running on GCE node
I20260929 18:42:23.473862 10208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:23.474951 10208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:23.476106 10208 hybrid_clock.cc:648] HybridClock initialized: now 1790707343476087 us; error 31 us; skew 500 ppm
Sep 29 18:42:23 dist-test-slave-smpn krb5kdc[10195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707343, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:23.478693 10208 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:42:23.479609 10208 webserver.cc:460] Webserver started at http://127.7.157.126:36427/ using document root <none> and password file <none>
I20260929 18:42:23.480110 10208 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:23.480175 10208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:23.480345 10208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:23.482357 10208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "036224ac22774ce3a6c9f492056ce7e5"
format_stamp: "Formatted at 2026-09-29 18:42:23 on dist-test-slave-smpn"
server_key: "5eea88679e7a0588a4bf5e839d114607"
server_key_iv: "52e23ae54297ee1a76f83a5f257e43bb"
server_key_version: "encryptionkey@0"
I20260929 18:42:23.482926 10208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "036224ac22774ce3a6c9f492056ce7e5"
format_stamp: "Formatted at 2026-09-29 18:42:23 on dist-test-slave-smpn"
server_key: "5eea88679e7a0588a4bf5e839d114607"
server_key_iv: "52e23ae54297ee1a76f83a5f257e43bb"
server_key_version: "encryptionkey@0"
I20260929 18:42:23.486680 10208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:42:23.489100 10223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:23.490300 10208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:23.490399 10208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "036224ac22774ce3a6c9f492056ce7e5"
format_stamp: "Formatted at 2026-09-29 18:42:23 on dist-test-slave-smpn"
server_key: "5eea88679e7a0588a4bf5e839d114607"
server_key_iv: "52e23ae54297ee1a76f83a5f257e43bb"
server_key_version: "encryptionkey@0"
I20260929 18:42:23.490487 10208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:23.499462 10208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:23.502002 10208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:23.502168 10208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:23.508783 10208 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:34891
I20260929 18:42:23.508798 10275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:34891 every 8 connection(s)
I20260929 18:42:23.509708 10208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:23.511826 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 10208
I20260929 18:42:23.511935 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:23.512213 7797 external_mini_cluster.cc:1428] Setting key 74c0a24db4502fa28e9574a9b73b6c2d
I20260929 18:42:23.512691 10276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:23.519339 10276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5: Bootstrap starting.
Sep 29 18:42:23 dist-test-slave-smpn krb5kdc[10195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:23.522037 10276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:23.522917 10276 log.cc:824] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:23.525207 10276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5: No bootstrap required, opened a new log
I20260929 18:42:23.526489 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:23.513531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.7.157.126:34891)
0929 18:42:23.513930 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:23.513948 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:23.513983 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:42:23.514614 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:23.514633 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:23.514952 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:23.515285 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:23.516140 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:23.517145 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:23.518297 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:23.518771 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:23.518884 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:23.521600 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:23.521605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:23.521621 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:23.521652 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:23.523517 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:23.523951 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:23.523956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:23.523963 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:23.524094 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:23.524317 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:23.524319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:23.524321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:23.524853 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:23.525044 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:23.525218 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 18:42:23.525522 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":111,"threads_started":1}
I20260929 18:42:23.529378 10276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036224ac22774ce3a6c9f492056ce7e5" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 34891 } }
I20260929 18:42:23.529598 10276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:23.529642 10276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036224ac22774ce3a6c9f492056ce7e5, State: Initialized, Role: FOLLOWER
I20260929 18:42:23.530002 10276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036224ac22774ce3a6c9f492056ce7e5" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 34891 } }
I20260929 18:42:23.530108 10276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:23.530159 10276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:23.530228 10276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:23.531112 10276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036224ac22774ce3a6c9f492056ce7e5" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 34891 } }
I20260929 18:42:23.531417 10276 leader_election.cc:302] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 036224ac22774ce3a6c9f492056ce7e5; no voters:
I20260929 18:42:23.531669 10276 leader_election.cc:288] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:23.531782 10281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:23.531978 10281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [term 1 LEADER]: Becoming Leader. State: Replica: 036224ac22774ce3a6c9f492056ce7e5, State: Running, Role: LEADER
I20260929 18:42:23.532307 10281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036224ac22774ce3a6c9f492056ce7e5" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 34891 } }
I20260929 18:42:23.532768 10276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:23.533840 10283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 036224ac22774ce3a6c9f492056ce7e5. Latest consensus state: current_term: 1 leader_uuid: "036224ac22774ce3a6c9f492056ce7e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036224ac22774ce3a6c9f492056ce7e5" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 34891 } } }
I20260929 18:42:23.533967 10283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:23.534369 10290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:23.534505 10282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "036224ac22774ce3a6c9f492056ce7e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036224ac22774ce3a6c9f492056ce7e5" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 34891 } } }
I20260929 18:42:23.534709 10282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:23.537317 10290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:23.542548 10290 catalog_manager.cc:1347] Generated new cluster ID: 34788e65cf264621b450e643facb23b4
I20260929 18:42:23.542615 10290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:23.556144 10290 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:23.557363 10290 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:23.564819 10290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 036224ac22774ce3a6c9f492056ce7e5: Generated new TSK 0
I20260929 18:42:23.565551 10290 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:42:23.598965 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:34891
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:23.709378 10302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:23.709649 10302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:23.709692 10302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:23.713636 10302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:23.713702 10302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:23.713774 10302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:42:23.718997 10302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:34891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:23.720048 10302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:23.721060 10302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:23.730151 10310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:23.730342 10308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:23.730201 10307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:23.730278 10302 server_base.cc:1034] running on GCE node
I20260929 18:42:23.730786 10302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:23.731426 10302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:23.732590 10302 hybrid_clock.cc:648] HybridClock initialized: now 1790707343732571 us; error 33 us; skew 500 ppm
Sep 29 18:42:23 dist-test-slave-smpn krb5kdc[10195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707343, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:23.735345 10302 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:42:23.736232 10302 webserver.cc:460] Webserver started at http://127.7.157.65:39155/ using document root <none> and password file <none>
I20260929 18:42:23.736778 10302 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:23.736841 10302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:23.737027 10302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:23.739118 10302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "d7fc8819ad274ed8b15338d864a3f7d6"
format_stamp: "Formatted at 2026-09-29 18:42:23 on dist-test-slave-smpn"
server_key: "07910a661054e963b9af278b161e76f2"
server_key_iv: "c21272961fe9301bedf999f1583125b4"
server_key_version: "encryptionkey@0"
I20260929 18:42:23.739691 10302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "d7fc8819ad274ed8b15338d864a3f7d6"
format_stamp: "Formatted at 2026-09-29 18:42:23 on dist-test-slave-smpn"
server_key: "07910a661054e963b9af278b161e76f2"
server_key_iv: "c21272961fe9301bedf999f1583125b4"
server_key_version: "encryptionkey@0"
I20260929 18:42:23.743376 10302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:42:23.745839 10317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:23.747015 10302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:23.747153 10302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "d7fc8819ad274ed8b15338d864a3f7d6"
format_stamp: "Formatted at 2026-09-29 18:42:23 on dist-test-slave-smpn"
server_key: "07910a661054e963b9af278b161e76f2"
server_key_iv: "c21272961fe9301bedf999f1583125b4"
server_key_version: "encryptionkey@0"
I20260929 18:42:23.747247 10302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:23.758662 10302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:23.761369 10302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:23.761547 10302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:23.762097 10302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:23.762897 10302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:23.762941 10302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:23.762992 10302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:23.763021 10302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:23.771387 10302 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:33665
I20260929 18:42:23.771411 10430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:33665 every 8 connection(s)
I20260929 18:42:23.772294 10302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:42:23.775242 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 10302
I20260929 18:42:23.775347 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:42:23.775592 7797 external_mini_cluster.cc:1428] Setting key 2dbb204c3a7ec34993850da13c345cd8
Sep 29 18:42:23 dist-test-slave-smpn krb5kdc[10195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707343, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
I20260929 18:42:23.785905 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:23.774038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:58653 (local address 127.7.157.126:34891)
0929 18:42:23.774176 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:23.774180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:23.775109 (+ 929us) server_negotiation.cc:407] Connection header received
0929 18:42:23.776466 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:23.776469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:23.776528 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:23.776601 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:23.777833 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:23.778445 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:23.779655 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:23.779984 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:23.780087 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:23.782471 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:23.782475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:23.782485 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:23.782514 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:23.783987 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:23.784503 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:23.784506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:23.784508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:23.784594 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:23.784927 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:23.784928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:23.784929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:23.785091 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:23.785170 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:23.785679 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 18:42:23.785784 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
I20260929 18:42:23.786867 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:23.774482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:34891 (local address 127.7.157.65:58653)
0929 18:42:23.774914 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:23.774958 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:23.776274 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:23.776777 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:23.776785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:23.777218 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:23.777700 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:23.777713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:23.778585 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:23.778588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:23.779543 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:23.779553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:23.780145 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:23.780148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:23.780290 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:23.780896 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:23.780912 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:23.782302 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:23.784112 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:23.784117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:23.784130 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:23.784404 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:23.784699 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:23.784702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:23.784703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:23.784849 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:23.785197 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:23.785203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:23.785463 (+ 260us) client_negotiation.cc:769] Sending connection context
0929 18:42:23.785625 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 18:42:23.785903 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":118,"threads_started":1}
I20260929 18:42:23.788162 10431 heartbeater.cc:347] Connected to a master server at 127.7.157.126:34891
I20260929 18:42:23.788434 10431 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:23.789003 10431 heartbeater.cc:511] Master 127.7.157.126:34891 requested a full tablet report, sending...
I20260929 18:42:23.790534 10240 ts_manager.cc:194] Registered new tserver with Master: d7fc8819ad274ed8b15338d864a3f7d6 (127.7.157.65:33665)
I20260929 18:42:23.791706 10240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:58653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:42:23.801345 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:34891
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:23.909155 10436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:23.909442 10436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:23.909505 10436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:23.913584 10436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:23.913661 10436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:23.913743 10436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:42:23.919145 10436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:34891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:23.920270 10436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:23.921315 10436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:23.930626 10441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:23.930686 10442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:23.930732 10436 server_base.cc:1034] running on GCE node
W20260929 18:42:23.930652 10444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:23.931252 10436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:23.931852 10436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:23.933008 10436 hybrid_clock.cc:648] HybridClock initialized: now 1790707343932990 us; error 35 us; skew 500 ppm
Sep 29 18:42:23 dist-test-slave-smpn krb5kdc[10195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707343, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:23.935734 10436 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:42:23.936617 10436 webserver.cc:460] Webserver started at http://127.7.157.66:41033/ using document root <none> and password file <none>
I20260929 18:42:23.937165 10436 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:23.937228 10436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:23.937417 10436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:23.939515 10436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "636c446ae4444f4dbca8161b00013ab7"
format_stamp: "Formatted at 2026-09-29 18:42:23 on dist-test-slave-smpn"
server_key: "740c5ee26181a7315e7900d27c2a0d4e"
server_key_iv: "74977b6844abe28e0973d813f1543568"
server_key_version: "encryptionkey@0"
I20260929 18:42:23.940063 10436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "636c446ae4444f4dbca8161b00013ab7"
format_stamp: "Formatted at 2026-09-29 18:42:23 on dist-test-slave-smpn"
server_key: "740c5ee26181a7315e7900d27c2a0d4e"
server_key_iv: "74977b6844abe28e0973d813f1543568"
server_key_version: "encryptionkey@0"
I20260929 18:42:23.943949 10436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:42:23.946374 10451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:23.947726 10436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:23.947829 10436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "636c446ae4444f4dbca8161b00013ab7"
format_stamp: "Formatted at 2026-09-29 18:42:23 on dist-test-slave-smpn"
server_key: "740c5ee26181a7315e7900d27c2a0d4e"
server_key_iv: "74977b6844abe28e0973d813f1543568"
server_key_version: "encryptionkey@0"
I20260929 18:42:23.947932 10436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:23.961462 10436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:23.964180 10436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:23.964340 10436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:23.964896 10436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:23.965685 10436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:23.965731 10436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:23.965773 10436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:23.965801 10436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:23.974050 10436 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:35249
I20260929 18:42:23.974076 10564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:35249 every 8 connection(s)
I20260929 18:42:23.974922 10436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:42:23.976702 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 10436
I20260929 18:42:23.976811 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:42:23.977082 7797 external_mini_cluster.cc:1428] Setting key 5e2674c84bab8d1b74532af856002764
Sep 29 18:42:23 dist-test-slave-smpn krb5kdc[10195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707343, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
I20260929 18:42:23.989817 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:23.976855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:57569 (local address 127.7.157.126:34891)
0929 18:42:23.976988 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:23.976994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:23.978038 (+ 1044us) server_negotiation.cc:407] Connection header received
0929 18:42:23.979845 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:23.979848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:23.979890 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:23.979949 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:23.981335 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:23.982096 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:23.983473 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:23.983859 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:23.983981 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:23.986433 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:23.986437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:23.986440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:23.986470 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:23.987815 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:23.988274 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:23.988276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:23.988277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:23.988344 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:23.988725 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:23.988729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:23.988731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:23.988943 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:23.989021 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:23.989548 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 18:42:23.989684 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:42:23.990896 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:23.977309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:34891 (local address 127.7.157.66:57569)
0929 18:42:23.977884 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:23.977937 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:23.979591 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:23.980141 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:23.980150 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:23.980679 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:23.981189 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:23.981203 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:23.982240 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:23.982243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:23.983301 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:23.983311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:23.984043 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:23.984046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:23.984202 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:23.984813 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:23.984829 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:23.986251 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:23.987922 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:23.987927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:23.987942 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:23.988193 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:23.988438 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:23.988441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:23.988443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:23.988587 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:23.989028 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:23.989033 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:23.989288 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 18:42:23.989489 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 18:42:23.989826 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":137,"threads_started":1}
I20260929 18:42:23.992095 10565 heartbeater.cc:347] Connected to a master server at 127.7.157.126:34891
I20260929 18:42:23.992354 10565 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:23.992923 10565 heartbeater.cc:511] Master 127.7.157.126:34891 requested a full tablet report, sending...
I20260929 18:42:23.994094 10240 ts_manager.cc:194] Registered new tserver with Master: 636c446ae4444f4dbca8161b00013ab7 (127.7.157.66:35249)
I20260929 18:42:23.994917 10240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:57569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:42:24.005290 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:34891
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:24.119463 10570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:24.119864 10570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:24.119948 10570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:24.125334 10570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:24.125424 10570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:24.125515 10570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:42:24.132339 10570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:34891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:24.133869 10570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:24.135390 10570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:24.148203 10578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:24.148262 10576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:24.148487 10575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:24.148528 10570 server_base.cc:1034] running on GCE node
I20260929 18:42:24.148967 10570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:24.149674 10570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:24.150825 10570 hybrid_clock.cc:648] HybridClock initialized: now 1790707344150802 us; error 41 us; skew 500 ppm
Sep 29 18:42:24 dist-test-slave-smpn krb5kdc[10195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707344, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:24.153652 10570 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:42:24.154664 10570 webserver.cc:460] Webserver started at http://127.7.157.67:38945/ using document root <none> and password file <none>
I20260929 18:42:24.155300 10570 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:24.155373 10570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:24.155575 10570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:24.157661 10570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "90e4312215f445bd8d4652cf8954666c"
format_stamp: "Formatted at 2026-09-29 18:42:24 on dist-test-slave-smpn"
server_key: "da64f1969d09f72d9ac5ef138be09123"
server_key_iv: "fd14733b62e5f7b431c9aa84d2d5718d"
server_key_version: "encryptionkey@0"
I20260929 18:42:24.158232 10570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "90e4312215f445bd8d4652cf8954666c"
format_stamp: "Formatted at 2026-09-29 18:42:24 on dist-test-slave-smpn"
server_key: "da64f1969d09f72d9ac5ef138be09123"
server_key_iv: "fd14733b62e5f7b431c9aa84d2d5718d"
server_key_version: "encryptionkey@0"
I20260929 18:42:24.162050 10570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:42:24.164654 10585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:24.165956 10570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:24.166077 10570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "90e4312215f445bd8d4652cf8954666c"
format_stamp: "Formatted at 2026-09-29 18:42:24 on dist-test-slave-smpn"
server_key: "da64f1969d09f72d9ac5ef138be09123"
server_key_iv: "fd14733b62e5f7b431c9aa84d2d5718d"
server_key_version: "encryptionkey@0"
I20260929 18:42:24.166194 10570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:24.193524 10570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:24.196508 10570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:24.196693 10570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:24.197299 10570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:24.198114 10570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:24.198169 10570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:24.198221 10570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:24.198251 10570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:24.206638 10570 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:43387
I20260929 18:42:24.206691 10698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:43387 every 8 connection(s)
I20260929 18:42:24.207639 10570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:42:24.211915 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 10570
I20260929 18:42:24.212013 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:42:24.212265 7797 external_mini_cluster.cc:1428] Setting key f04edbbcb723dd07b0efc539a1cabb09
Sep 29 18:42:24 dist-test-slave-smpn krb5kdc[10195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707344, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:24.220834 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:24.209501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:58711 (local address 127.7.157.126:34891)
0929 18:42:24.209627 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:24.209644 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:24.210513 (+ 869us) server_negotiation.cc:407] Connection header received
0929 18:42:24.211829 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:24.211833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:24.211891 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:24.211965 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:24.213405 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:24.213943 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:24.215111 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:24.215334 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:24.215420 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:24.217722 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:24.217725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:24.217727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:24.217752 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:24.218897 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:24.219420 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:24.219423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:24.219425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:24.219500 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:24.219798 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:24.219801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:24.219803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:24.219978 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:24.220062 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:24.220545 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 18:42:24.220674 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:42:24.221649 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:24.209917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:34891 (local address 127.7.157.67:58711)
0929 18:42:24.210366 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:24.210410 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:24.211605 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:24.212165 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:24.212173 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:24.212585 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:24.213266 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:24.213284 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:24.214065 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:24.214068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:24.214968 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:24.214977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:24.215474 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:24.215477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:24.215656 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:24.216157 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:24.216174 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:24.217556 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:24.219000 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:24.219006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:24.219023 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:24.219329 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:24.219577 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:24.219579 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:24.219581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:24.219717 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:24.220071 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:24.220077 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:24.220327 (+ 250us) client_negotiation.cc:769] Sending connection context
0929 18:42:24.220493 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 18:42:24.220819 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":117,"threads_started":1}
I20260929 18:42:24.222721 10699 heartbeater.cc:347] Connected to a master server at 127.7.157.126:34891
I20260929 18:42:24.222963 10699 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:24.223557 10699 heartbeater.cc:511] Master 127.7.157.126:34891 requested a full tablet report, sending...
I20260929 18:42:24.224635 10240 ts_manager.cc:194] Registered new tserver with Master: 90e4312215f445bd8d4652cf8954666c (127.7.157.67:43387)
I20260929 18:42:24.225196 10240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:58711
I20260929 18:42:24.225901 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:42:24.237800 10279 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:24.236468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.7.157.126:34891)
0929 18:42:24.236583 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:24.236587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:24.236749 (+ 162us) server_negotiation.cc:407] Connection header received
0929 18:42:24.236863 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:24.236866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:24.236908 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:24.236982 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:24.237668 (+ 686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39378: BlockingRecv error: recv got EOF from 127.0.0.1:39378 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:42:24.238878 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 10302
I20260929 18:42:24.243753 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 10436
I20260929 18:42:24.248123 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 10570
I20260929 18:42:24.252482 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 10208
2026-09-29T18:42:24Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1987 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:42:24 dist-test-slave-smpn krb5kdc[10711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:42:24 dist-test-slave-smpn krb5kdc[10711](info): set up 2 sockets
Sep 29 18:42:24 dist-test-slave-smpn krb5kdc[10711](info): commencing operation
krb5kdc: starting...
W20260929 18:42:25.210824 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:42:25 dist-test-slave-smpn krb5kdc[10711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:42:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:42:25.329545 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:45733
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:33943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:45733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:25.439245 10724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:25.439546 10724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:25.439603 10724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:25.443813 10724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:25.443889 10724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:25.443915 10724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:25.443938 10724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:25.443969 10724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:25.449460 10724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:33943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:45733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:45733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:25.450600 10724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:25.451692 10724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:25.460191 10729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:25.460250 10732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:25.460273 10730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:25.460405 10724 server_base.cc:1034] running on GCE node
I20260929 18:42:25.461097 10724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:25.462319 10724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:25.463479 10724 hybrid_clock.cc:648] HybridClock initialized: now 1790707345463459 us; error 35 us; skew 500 ppm
Sep 29 18:42:25 dist-test-slave-smpn krb5kdc[10711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707345, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:25.469329 10724 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:42:25.470283 10724 webserver.cc:460] Webserver started at http://127.7.157.126:39585/ using document root <none> and password file <none>
I20260929 18:42:25.470853 10724 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:25.470916 10724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:25.471128 10724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:25.473234 10724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "08da74df0f5040348da61bfd098eacd2"
format_stamp: "Formatted at 2026-09-29 18:42:25 on dist-test-slave-smpn"
server_key: "cbae77bd3f5d2e2ec0cd4b67b056c6d5"
server_key_iv: "d72840b95e1a8be3fabd0e3f117c31c3"
server_key_version: "encryptionkey@0"
I20260929 18:42:25.473767 10724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "08da74df0f5040348da61bfd098eacd2"
format_stamp: "Formatted at 2026-09-29 18:42:25 on dist-test-slave-smpn"
server_key: "cbae77bd3f5d2e2ec0cd4b67b056c6d5"
server_key_iv: "d72840b95e1a8be3fabd0e3f117c31c3"
server_key_version: "encryptionkey@0"
I20260929 18:42:25.477519 10724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 18:42:25.479970 10739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:25.481231 10724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:25.481340 10724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "08da74df0f5040348da61bfd098eacd2"
format_stamp: "Formatted at 2026-09-29 18:42:25 on dist-test-slave-smpn"
server_key: "cbae77bd3f5d2e2ec0cd4b67b056c6d5"
server_key_iv: "d72840b95e1a8be3fabd0e3f117c31c3"
server_key_version: "encryptionkey@0"
I20260929 18:42:25.481442 10724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:25.492527 10724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:25.495580 10724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:25.495760 10724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:25.502271 10724 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:45733
I20260929 18:42:25.502523 10791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:45733 every 8 connection(s)
I20260929 18:42:25.503247 10724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:25.505121 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 10724
I20260929 18:42:25.505224 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:25.505471 7797 external_mini_cluster.cc:1428] Setting key e1845d9715770404eae7614d9a7cecff
I20260929 18:42:25.506129 10792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:25.512595 10792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2: Bootstrap starting.
Sep 29 18:42:25 dist-test-slave-smpn krb5kdc[10711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:25.515156 10792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:25.515933 10792 log.cc:824] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:25.517743 10792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2: No bootstrap required, opened a new log
I20260929 18:42:25.518963 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:25.506714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.7.157.126:45733)
0929 18:42:25.507152 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:25.507168 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:25.507204 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:42:25.507845 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:25.507862 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:25.508190 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:25.508471 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:25.509361 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:25.510346 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:25.511511 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:25.511973 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:25.512083 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:25.514601 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:25.514606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:25.514619 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:25.514649 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:25.516289 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:25.516689 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:25.516694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:25.516700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:25.516808 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:25.517053 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:25.517057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:25.517058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:25.517457 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:25.517636 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:25.517914 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 18:42:25.518250 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":124,"threads_started":1}
I20260929 18:42:25.521445 10792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08da74df0f5040348da61bfd098eacd2" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45733 } }
I20260929 18:42:25.521703 10792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:25.521749 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:42:25.521766 10792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08da74df0f5040348da61bfd098eacd2, State: Initialized, Role: FOLLOWER
I20260929 18:42:25.522178 10792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08da74df0f5040348da61bfd098eacd2" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45733 } }
I20260929 18:42:25.522300 10792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:25.522358 10792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:25.522447 10792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:25.523422 10792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08da74df0f5040348da61bfd098eacd2" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45733 } }
I20260929 18:42:25.523813 10792 leader_election.cc:302] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08da74df0f5040348da61bfd098eacd2; no voters:
I20260929 18:42:25.524138 10792 leader_election.cc:288] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:25.524238 10798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:25.524453 10798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [term 1 LEADER]: Becoming Leader. State: Replica: 08da74df0f5040348da61bfd098eacd2, State: Running, Role: LEADER
I20260929 18:42:25.524804 10798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08da74df0f5040348da61bfd098eacd2" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45733 } }
I20260929 18:42:25.525453 10792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:25.525902 10798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08da74df0f5040348da61bfd098eacd2. Latest consensus state: current_term: 1 leader_uuid: "08da74df0f5040348da61bfd098eacd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08da74df0f5040348da61bfd098eacd2" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45733 } } }
I20260929 18:42:25.526100 10798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:25.526315 10799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08da74df0f5040348da61bfd098eacd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08da74df0f5040348da61bfd098eacd2" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45733 } } }
I20260929 18:42:25.526456 10799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:25.526520 10803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:25.529788 10803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:25.536109 10803 catalog_manager.cc:1347] Generated new cluster ID: 83224b7adf044e7d85e406cbede5093d
I20260929 18:42:25.536195 10803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:25.545780 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:25.540201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.7.157.126:45733)
0929 18:42:25.540306 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:25.540310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:25.540566 (+ 256us) server_negotiation.cc:407] Connection header received
0929 18:42:25.540687 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:25.540691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:25.540747 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:25.540835 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:25.541383 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:25.542226 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:25.543374 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:25.543760 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:25.543862 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:25.544706 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:25.544710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:25.544713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:42:25.545324 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:25.545362 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:42:25.545383 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:25.545472 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:25.545551 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 18:42:25.545619 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":38}
I20260929 18:42:25.548256 10803 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:25.549139 10803 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:25.555648 10803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08da74df0f5040348da61bfd098eacd2: Generated new TSK 0
I20260929 18:42:25.556391 10803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:42:25.565255 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 10724
2026-09-29T18:42:25Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1311 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:42:25 dist-test-slave-smpn krb5kdc[10825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:42:25 dist-test-slave-smpn krb5kdc[10825](info): set up 2 sockets
Sep 29 18:42:25 dist-test-slave-smpn krb5kdc[10825](info): commencing operation
krb5kdc: starting...
W20260929 18:42:26.526814 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:42:26 dist-test-slave-smpn krb5kdc[10825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:42:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:42:26.642606 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:45625
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:45625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:26.751209 10838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:26.751498 10838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:26.751554 10838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:26.755540 10838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:26.755605 10838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:26.755631 10838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:26.755658 10838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:26.755692 10838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:26.760991 10838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:45625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:45625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:26.762074 10838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:26.763170 10838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:26.771306 10846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:26.771309 10844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:26.771355 10838 server_base.cc:1034] running on GCE node
W20260929 18:42:26.771309 10843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:26.772035 10838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:26.773165 10838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:26.774317 10838 hybrid_clock.cc:648] HybridClock initialized: now 1790707346774293 us; error 40 us; skew 500 ppm
Sep 29 18:42:26 dist-test-slave-smpn krb5kdc[10825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707346, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:26.776913 10838 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:42:26.777782 10838 webserver.cc:460] Webserver started at http://127.7.157.126:37621/ using document root <none> and password file <none>
I20260929 18:42:26.778312 10838 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:26.778371 10838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:26.778561 10838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:26.780579 10838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "1ab796181ebe4f77b8d371a9dc6b3bbf"
format_stamp: "Formatted at 2026-09-29 18:42:26 on dist-test-slave-smpn"
server_key: "2046e2e1e8c06d1468244f8ee6b3f147"
server_key_iv: "db8a7f07a49035ec7f0cec36e39d091e"
server_key_version: "encryptionkey@0"
I20260929 18:42:26.781108 10838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "1ab796181ebe4f77b8d371a9dc6b3bbf"
format_stamp: "Formatted at 2026-09-29 18:42:26 on dist-test-slave-smpn"
server_key: "2046e2e1e8c06d1468244f8ee6b3f147"
server_key_iv: "db8a7f07a49035ec7f0cec36e39d091e"
server_key_version: "encryptionkey@0"
I20260929 18:42:26.784814 10838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:42:26.787120 10853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:26.788369 10838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:26.788475 10838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "1ab796181ebe4f77b8d371a9dc6b3bbf"
format_stamp: "Formatted at 2026-09-29 18:42:26 on dist-test-slave-smpn"
server_key: "2046e2e1e8c06d1468244f8ee6b3f147"
server_key_iv: "db8a7f07a49035ec7f0cec36e39d091e"
server_key_version: "encryptionkey@0"
I20260929 18:42:26.788574 10838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:26.807400 10838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:26.810352 10838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:26.810521 10838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:26.817538 10838 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:45625
I20260929 18:42:26.817826 10905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:45625 every 8 connection(s)
I20260929 18:42:26.818850 10838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:26.822232 10906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:26.828109 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 10838
I20260929 18:42:26.828223 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:26.828464 7797 external_mini_cluster.cc:1428] Setting key 0a6cc8cbc2ea473e420e65a4cc99db6d
I20260929 18:42:26.829146 10906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf: Bootstrap starting.
I20260929 18:42:26.831861 10906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:26.832739 10906 log.cc:824] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:26.834795 10906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf: No bootstrap required, opened a new log
Sep 29 18:42:26 dist-test-slave-smpn krb5kdc[10825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:26.838554 10906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab796181ebe4f77b8d371a9dc6b3bbf" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45625 } }
I20260929 18:42:26.838887 10906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:26.838949 10906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ab796181ebe4f77b8d371a9dc6b3bbf, State: Initialized, Role: FOLLOWER
I20260929 18:42:26.839433 10906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab796181ebe4f77b8d371a9dc6b3bbf" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45625 } }
I20260929 18:42:26.839567 10906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:26.839632 10906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:26.839728 10906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:26.840585 10906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab796181ebe4f77b8d371a9dc6b3bbf" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45625 } }
I20260929 18:42:26.840939 10906 leader_election.cc:302] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ab796181ebe4f77b8d371a9dc6b3bbf; no voters:
I20260929 18:42:26.841384 10906 leader_election.cc:288] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:26.841449 10911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:26.841678 10911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [term 1 LEADER]: Becoming Leader. State: Replica: 1ab796181ebe4f77b8d371a9dc6b3bbf, State: Running, Role: LEADER
I20260929 18:42:26.841979 10911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab796181ebe4f77b8d371a9dc6b3bbf" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45625 } }
I20260929 18:42:26.842650 10906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:26.842952 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:26.829839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.7.157.126:45625)
0929 18:42:26.830271 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:26.830292 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:26.830337 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:42:26.831196 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:26.831221 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:26.831665 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:26.832057 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:26.832700 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:26.834030 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:26.834862 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:26.835565 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:26.835741 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:26.837959 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:26.837966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:26.837983 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:26.838016 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:26.839740 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:26.840136 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:26.840142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:26.840161 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:26.840279 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:26.840534 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:26.840537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:26.840539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:26.841091 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:26.841293 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:26.841623 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 18:42:26.841952 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":105,"threads_started":1}
I20260929 18:42:26.843255 10911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ab796181ebe4f77b8d371a9dc6b3bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab796181ebe4f77b8d371a9dc6b3bbf" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45625 } } }
I20260929 18:42:26.843374 10911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:26.843562 10911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ab796181ebe4f77b8d371a9dc6b3bbf. Latest consensus state: current_term: 1 leader_uuid: "1ab796181ebe4f77b8d371a9dc6b3bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab796181ebe4f77b8d371a9dc6b3bbf" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45625 } } }
I20260929 18:42:26.843649 10911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf [sys.catalog]: This master's current role is: LEADER
W20260929 18:42:26.847450 10926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:42:26.847538 10926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:42:26.847609 10917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:26.848465 10917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:26.854104 10917 catalog_manager.cc:1347] Generated new cluster ID: 71a2b14d514c44cb9e92f65f02e3e9bc
I20260929 18:42:26.854185 10917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:26.869743 10917 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:26.870823 10917 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:26.874568 10917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ab796181ebe4f77b8d371a9dc6b3bbf: Generated new TSK 0
I20260929 18:42:26.875295 10917 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:42:26.906036 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45625
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:27.016544 10932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:27.017061 10932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:27.017172 10932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:27.021552 10932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:27.021610 10932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:27.021694 10932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:42:27.027045 10932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:27.028191 10932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:27.029225 10932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:42:27.038618 10932 server_base.cc:1034] running on GCE node
W20260929 18:42:27.038569 10937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:27.038684 10940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:27.038563 10938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:27.039170 10932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:27.039760 10932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:27.040905 10932 hybrid_clock.cc:648] HybridClock initialized: now 1790707347040881 us; error 37 us; skew 500 ppm
Sep 29 18:42:27 dist-test-slave-smpn krb5kdc[10825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707347, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:27.043677 10932 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:42:27.044559 10932 webserver.cc:460] Webserver started at http://127.7.157.65:38075/ using document root <none> and password file <none>
I20260929 18:42:27.045120 10932 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:27.045188 10932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:27.045382 10932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:27.047402 10932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "e2f591a309f44aa897fda184464a78ea"
format_stamp: "Formatted at 2026-09-29 18:42:27 on dist-test-slave-smpn"
server_key: "a49cf751097a07db9a5494d6907287b8"
server_key_iv: "006a08e258e12a5b2b6af9919886d1d6"
server_key_version: "encryptionkey@0"
I20260929 18:42:27.047928 10932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "e2f591a309f44aa897fda184464a78ea"
format_stamp: "Formatted at 2026-09-29 18:42:27 on dist-test-slave-smpn"
server_key: "a49cf751097a07db9a5494d6907287b8"
server_key_iv: "006a08e258e12a5b2b6af9919886d1d6"
server_key_version: "encryptionkey@0"
I20260929 18:42:27.051479 10932 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:42:27.053803 10947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:27.055019 10932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:27.055179 10932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "e2f591a309f44aa897fda184464a78ea"
format_stamp: "Formatted at 2026-09-29 18:42:27 on dist-test-slave-smpn"
server_key: "a49cf751097a07db9a5494d6907287b8"
server_key_iv: "006a08e258e12a5b2b6af9919886d1d6"
server_key_version: "encryptionkey@0"
I20260929 18:42:27.055274 10932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:27.069664 10932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:27.072419 10932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:27.072594 10932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:27.073195 10932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:27.074036 10932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:27.074083 10932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:27.074136 10932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:27.074162 10932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:27.082386 10932 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:39751
I20260929 18:42:27.082419 11060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:39751 every 8 connection(s)
I20260929 18:42:27.083287 10932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:42:27.092590 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 10932
I20260929 18:42:27.092702 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
Sep 29 18:42:27 dist-test-slave-smpn krb5kdc[10825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707347, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:27.092979 7797 external_mini_cluster.cc:1428] Setting key 8eb6dd7b23502df1b07ebefcba58ad92
I20260929 18:42:27.097324 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:27.084962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:42817 (local address 127.7.157.126:45625)
0929 18:42:27.085097 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:27.085102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:27.086048 (+ 946us) server_negotiation.cc:407] Connection header received
0929 18:42:27.087533 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:27.087537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:27.087598 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:27.087685 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:27.088925 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:27.089466 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:27.090639 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:27.090880 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:27.090989 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:27.093547 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:27.093551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:27.093553 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:27.093579 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:27.095116 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:27.095722 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:27.095724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:27.095726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:27.095801 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:27.096215 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:27.096218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:27.096220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:27.096443 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:27.096534 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:27.097045 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 18:42:27.097173 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:42:27.098182 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:27.085395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45625 (local address 127.7.157.65:42817)
0929 18:42:27.085862 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:27.085906 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:27.087299 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:27.087874 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:27.087881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:27.088255 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:27.088760 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:27.088774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:27.089597 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:27.089600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:27.090501 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:27.090519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:27.091040 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:27.091044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:27.091242 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:27.091796 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:27.091812 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:27.093382 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:27.095232 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:27.095240 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:27.095257 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:27.095626 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:27.095880 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:27.095883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:27.095885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:27.096106 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:27.096536 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:27.096542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:27.096796 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 18:42:27.096980 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 18:42:27.097258 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":113,"threads_started":1}
I20260929 18:42:27.099571 11061 heartbeater.cc:347] Connected to a master server at 127.7.157.126:45625
I20260929 18:42:27.099818 11061 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:27.100306 11061 heartbeater.cc:511] Master 127.7.157.126:45625 requested a full tablet report, sending...
I20260929 18:42:27.102066 10870 ts_manager.cc:194] Registered new tserver with Master: e2f591a309f44aa897fda184464a78ea (127.7.157.65:39751)
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
I20260929 18:42:27.103654 10870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:42817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:42:27.122622 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45625
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:27.230460 11066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:27.230731 11066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:27.230772 11066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:27.234699 11066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:27.234760 11066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:27.234823 11066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:42:27.240057 11066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:27.241046 11066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:27.242091 11066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:27.251854 11072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:27.251816 11071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:27.251816 11074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:27.252103 11066 server_base.cc:1034] running on GCE node
I20260929 18:42:27.252645 11066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:27.253304 11066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:27.254520 11066 hybrid_clock.cc:648] HybridClock initialized: now 1790707347254500 us; error 35 us; skew 500 ppm
Sep 29 18:42:27 dist-test-slave-smpn krb5kdc[10825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707347, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:27.257520 11066 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:42:27.258507 11066 webserver.cc:460] Webserver started at http://127.7.157.66:42805/ using document root <none> and password file <none>
I20260929 18:42:27.259105 11066 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:27.259183 11066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:27.259375 11066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:27.261502 11066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "4b6694d6b5974bc4bc75f8724d6e7c09"
format_stamp: "Formatted at 2026-09-29 18:42:27 on dist-test-slave-smpn"
server_key: "b5e210f8507ef6fe9160eb49717dbcb3"
server_key_iv: "07a476f20d5d603bd28ee4d42774ac9f"
server_key_version: "encryptionkey@0"
I20260929 18:42:27.262063 11066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "4b6694d6b5974bc4bc75f8724d6e7c09"
format_stamp: "Formatted at 2026-09-29 18:42:27 on dist-test-slave-smpn"
server_key: "b5e210f8507ef6fe9160eb49717dbcb3"
server_key_iv: "07a476f20d5d603bd28ee4d42774ac9f"
server_key_version: "encryptionkey@0"
I20260929 18:42:27.266112 11066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 18:42:27.268748 11081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:27.270030 11066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:27.270135 11066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "4b6694d6b5974bc4bc75f8724d6e7c09"
format_stamp: "Formatted at 2026-09-29 18:42:27 on dist-test-slave-smpn"
server_key: "b5e210f8507ef6fe9160eb49717dbcb3"
server_key_iv: "07a476f20d5d603bd28ee4d42774ac9f"
server_key_version: "encryptionkey@0"
I20260929 18:42:27.270227 11066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:27.304481 11066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:27.308009 11066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:27.308182 11066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:27.308765 11066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:27.309562 11066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:27.309605 11066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:27.309644 11066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:27.309664 11066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:27.317660 11066 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:38039
I20260929 18:42:27.317698 11194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:38039 every 8 connection(s)
I20260929 18:42:27.318518 11066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:42:27.319031 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 11066
I20260929 18:42:27.319170 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:42:27.319428 7797 external_mini_cluster.cc:1428] Setting key 9fc83ad27a54dcd4bb4ac1635b579699
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
Sep 29 18:42:27 dist-test-slave-smpn krb5kdc[10825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707347, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
I20260929 18:42:27.332943 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:27.321638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:59851 (local address 127.7.157.126:45625)
0929 18:42:27.321792 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:27.321798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:27.321815 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:42:27.322754 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:27.322757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:27.322810 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:27.322877 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:27.324398 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:27.325014 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:27.326690 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:27.326898 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:27.326970 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:27.329584 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:27.329587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:27.329589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:27.329613 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:27.330923 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:27.331477 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:27.331481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:27.331483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:27.331573 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:27.331883 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:27.331886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:27.331888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:27.332070 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:27.332152 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:27.332653 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 18:42:27.332781 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:42:27.333819 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:27.320677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45625 (local address 127.7.157.66:59851)
0929 18:42:27.321137 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:27.321181 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:27.322508 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:27.323093 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:27.323104 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:27.323600 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:27.324263 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:27.324283 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:27.325141 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:27.325145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:27.326557 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:27.326568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:27.327039 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:27.327043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:27.327284 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:27.327860 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:27.327882 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:27.329413 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:27.331032 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:27.331038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:27.331054 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:27.331379 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:27.331653 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:27.331656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:27.331658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:27.331792 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:27.332162 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:27.332169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:27.332422 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 18:42:27.332587 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 18:42:27.332916 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":137,"threads_started":1}
I20260929 18:42:27.334932 11195 heartbeater.cc:347] Connected to a master server at 127.7.157.126:45625
I20260929 18:42:27.335229 11195 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:27.335839 11195 heartbeater.cc:511] Master 127.7.157.126:45625 requested a full tablet report, sending...
I20260929 18:42:27.337024 10870 ts_manager.cc:194] Registered new tserver with Master: 4b6694d6b5974bc4bc75f8724d6e7c09 (127.7.157.66:38039)
I20260929 18:42:27.337791 10870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:59851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:42:27.346138 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45625
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:27.453940 11200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:27.454207 11200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:27.454253 11200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:27.458230 11200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:27.458292 11200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:27.458364 11200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:42:27.463672 11200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:27.464677 11200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:27.465656 11200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:42:27.474817 11200 server_base.cc:1034] running on GCE node
W20260929 18:42:27.474818 11208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:27.474804 11205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:27.474799 11206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:27.475391 11200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:27.476008 11200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:27.477169 11200 hybrid_clock.cc:648] HybridClock initialized: now 1790707347477141 us; error 29 us; skew 500 ppm
Sep 29 18:42:27 dist-test-slave-smpn krb5kdc[10825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707347, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:27.479740 11200 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:42:27.480545 11200 webserver.cc:460] Webserver started at http://127.7.157.67:45055/ using document root <none> and password file <none>
I20260929 18:42:27.481047 11200 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:27.481102 11200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:27.481273 11200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:27.483304 11200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "56fc814fa4ff49bfa2f7cd305b79c9af"
format_stamp: "Formatted at 2026-09-29 18:42:27 on dist-test-slave-smpn"
server_key: "3ff6e3ea3c1e5da37a274f71ce73b55b"
server_key_iv: "e06b41a462744afd6d16bfe391ef04f8"
server_key_version: "encryptionkey@0"
I20260929 18:42:27.483815 11200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "56fc814fa4ff49bfa2f7cd305b79c9af"
format_stamp: "Formatted at 2026-09-29 18:42:27 on dist-test-slave-smpn"
server_key: "3ff6e3ea3c1e5da37a274f71ce73b55b"
server_key_iv: "e06b41a462744afd6d16bfe391ef04f8"
server_key_version: "encryptionkey@0"
I20260929 18:42:27.487399 11200 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:42:27.489640 11215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:27.490835 11200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:27.490936 11200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "56fc814fa4ff49bfa2f7cd305b79c9af"
format_stamp: "Formatted at 2026-09-29 18:42:27 on dist-test-slave-smpn"
server_key: "3ff6e3ea3c1e5da37a274f71ce73b55b"
server_key_iv: "e06b41a462744afd6d16bfe391ef04f8"
server_key_version: "encryptionkey@0"
I20260929 18:42:27.491037 11200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:27.508083 11200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:27.510738 11200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:27.510906 11200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:27.511516 11200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:27.512336 11200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:27.512382 11200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:27.512424 11200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:27.512452 11200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:27.520766 11200 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:35995
I20260929 18:42:27.520790 11328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:35995 every 8 connection(s)
I20260929 18:42:27.521672 11200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:42:27 dist-test-slave-smpn krb5kdc[10825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707347, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:27.531597 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 11200
I20260929 18:42:27.531708 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:42:27.532011 7797 external_mini_cluster.cc:1428] Setting key 15dcc9c016347789500d655be4599f71
I20260929 18:42:27.535138 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:27.523453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:37761 (local address 127.7.157.126:45625)
0929 18:42:27.523584 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:27.523588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:27.524427 (+ 839us) server_negotiation.cc:407] Connection header received
0929 18:42:27.525773 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:27.525777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:27.525840 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:27.525916 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:27.527145 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:27.527693 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:27.528832 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:27.529052 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:27.529144 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:27.531518 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:27.531521 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:27.531524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:27.531551 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:27.533049 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:27.533578 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:27.533582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:27.533583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:27.533674 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:27.534021 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:27.534024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:27.534026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:27.534226 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:27.534319 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:27.534821 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 18:42:27.534946 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:42:27.536038 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:27.523831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45625 (local address 127.7.157.67:37761)
0929 18:42:27.524272 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:27.524316 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:27.525544 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:27.526108 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:27.526115 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:27.526485 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:27.526991 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:27.527005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:27.527813 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:27.527816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:27.528706 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:27.528715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:27.529194 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:27.529199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:27.529373 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:27.529907 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:27.529924 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:27.531330 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:27.533149 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:27.533155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:27.533171 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:27.533489 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:27.533752 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:27.533755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:27.533757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:27.533938 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:27.534327 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:27.534334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:27.534591 (+ 257us) client_negotiation.cc:769] Sending connection context
0929 18:42:27.534771 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 18:42:27.535198 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
I20260929 18:42:27.537046 11329 heartbeater.cc:347] Connected to a master server at 127.7.157.126:45625
I20260929 18:42:27.537302 11329 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:27.537930 11329 heartbeater.cc:511] Master 127.7.157.126:45625 requested a full tablet report, sending...
I20260929 18:42:27.538944 10870 ts_manager.cc:194] Registered new tserver with Master: 56fc814fa4ff49bfa2f7cd305b79c9af (127.7.157.67:35995)
I20260929 18:42:27.539530 10870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:37761
I20260929 18:42:27.545718 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:42:27 dist-test-slave-smpn krb5kdc[10825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707347, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 18:42:27 dist-test-slave-smpn krb5kdc[10825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707347, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:27.569749 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:27.562042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.7.157.126:45625)
0929 18:42:27.562206 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:27.562211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:27.562322 (+ 111us) server_negotiation.cc:407] Connection header received
0929 18:42:27.562474 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:27.562477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:27.562521 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:27.562585 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:27.563333 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:27.563846 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:27.564659 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:27.564870 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:27.564954 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:27.567121 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:27.567125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:27.567127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:27.567159 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:27.568198 (+ 1039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:27.568561 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:27.568564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:27.568566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:27.568668 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:27.568906 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:27.568909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:27.568910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:27.569129 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:27.569255 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:27.569472 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 18:42:27.569595 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 18:42:27.570128 10870 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52230
I20260929 18:42:27.570884 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 10932
I20260929 18:42:27.575670 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 11066
I20260929 18:42:27.580083 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 11200
I20260929 18:42:27.584524 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 10838
2026-09-29T18:42:27Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2020 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:42:27 dist-test-slave-smpn krb5kdc[11341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:42:27 dist-test-slave-smpn krb5kdc[11341](info): set up 2 sockets
Sep 29 18:42:27 dist-test-slave-smpn krb5kdc[11341](info): commencing operation
krb5kdc: starting...
W20260929 18:42:28.535127 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:42:28 dist-test-slave-smpn krb5kdc[11341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:42:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:42:28.651713 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:46209
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:46209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:28.759868 11354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:28.760143 11354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:28.760211 11354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:28.764226 11354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:28.764287 11354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:28.764314 11354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:28.764335 11354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:28.764364 11354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:28.769613 11354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:46209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:46209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:28.770628 11354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:28.771631 11354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:28.779819 11360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:28.779798 11362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:28.779798 11359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:28.779996 11354 server_base.cc:1034] running on GCE node
I20260929 18:42:28.780728 11354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:28.781852 11354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:28.783020 11354 hybrid_clock.cc:648] HybridClock initialized: now 1790707348782996 us; error 36 us; skew 500 ppm
Sep 29 18:42:28 dist-test-slave-smpn krb5kdc[11341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707348, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:28.785787 11354 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:42:28.786703 11354 webserver.cc:460] Webserver started at http://127.7.157.126:33403/ using document root <none> and password file <none>
I20260929 18:42:28.787307 11354 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:28.787379 11354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:28.787585 11354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:28.789680 11354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "9f606ee619ff4731aedd2b9d81fb1ae1"
format_stamp: "Formatted at 2026-09-29 18:42:28 on dist-test-slave-smpn"
server_key: "36e44b7a59f42ef8f50fddf8f04f76a9"
server_key_iv: "b7bf90768e6383b701517d71b0bf7088"
server_key_version: "encryptionkey@0"
I20260929 18:42:28.790221 11354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "9f606ee619ff4731aedd2b9d81fb1ae1"
format_stamp: "Formatted at 2026-09-29 18:42:28 on dist-test-slave-smpn"
server_key: "36e44b7a59f42ef8f50fddf8f04f76a9"
server_key_iv: "b7bf90768e6383b701517d71b0bf7088"
server_key_version: "encryptionkey@0"
I20260929 18:42:28.793975 11354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 18:42:28.796388 11369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:28.797690 11354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:28.797808 11354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "9f606ee619ff4731aedd2b9d81fb1ae1"
format_stamp: "Formatted at 2026-09-29 18:42:28 on dist-test-slave-smpn"
server_key: "36e44b7a59f42ef8f50fddf8f04f76a9"
server_key_iv: "b7bf90768e6383b701517d71b0bf7088"
server_key_version: "encryptionkey@0"
I20260929 18:42:28.797907 11354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:28.821973 11354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:28.825273 11354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:28.825462 11354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:28.832168 11354 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:46209
I20260929 18:42:28.832195 11421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:46209 every 8 connection(s)
I20260929 18:42:28.833112 11354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:28.835819 11422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:28.837291 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 11354
I20260929 18:42:28.837404 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:28.837690 7797 external_mini_cluster.cc:1428] Setting key 1cce615073de04d2df25f7d2da655c83
I20260929 18:42:28.841775 11422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1: Bootstrap starting.
I20260929 18:42:28.844033 11422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:28.844743 11422 log.cc:824] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:42:28 dist-test-slave-smpn krb5kdc[11341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:28.846867 11422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1: No bootstrap required, opened a new log
I20260929 18:42:28.850251 11422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f606ee619ff4731aedd2b9d81fb1ae1" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46209 } }
I20260929 18:42:28.850486 11422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:28.850521 11422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f606ee619ff4731aedd2b9d81fb1ae1, State: Initialized, Role: FOLLOWER
I20260929 18:42:28.851002 11422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f606ee619ff4731aedd2b9d81fb1ae1" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46209 } }
I20260929 18:42:28.851150 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:28.839131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.7.157.126:46209)
0929 18:42:28.839551 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:28.839566 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:28.839602 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:42:28.840209 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:28.840226 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:28.840548 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:28.840852 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:28.841611 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:28.842616 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:28.843486 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:28.843956 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:28.844099 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:28.846736 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:28.846741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:28.846754 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:28.846780 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:28.848327 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:28.848730 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:28.848735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:28.848740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:28.848850 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:28.849099 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:28.849102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:28.849104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:28.849493 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:28.849697 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:28.849973 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 18:42:28.850258 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":128,"threads_started":1}
I20260929 18:42:28.851166 11422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:28.851249 11422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:28.851334 11422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:28.852257 11422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f606ee619ff4731aedd2b9d81fb1ae1" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46209 } }
I20260929 18:42:28.852581 11422 leader_election.cc:302] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f606ee619ff4731aedd2b9d81fb1ae1; no voters:
I20260929 18:42:28.852875 11422 leader_election.cc:288] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:28.852967 11427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:28.853168 11427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [term 1 LEADER]: Becoming Leader. State: Replica: 9f606ee619ff4731aedd2b9d81fb1ae1, State: Running, Role: LEADER
I20260929 18:42:28.853492 11427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f606ee619ff4731aedd2b9d81fb1ae1" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46209 } }
I20260929 18:42:28.854208 11422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:28.854683 11427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f606ee619ff4731aedd2b9d81fb1ae1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f606ee619ff4731aedd2b9d81fb1ae1" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46209 } } }
I20260929 18:42:28.854887 11427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:28.855172 11428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f606ee619ff4731aedd2b9d81fb1ae1. Latest consensus state: current_term: 1 leader_uuid: "9f606ee619ff4731aedd2b9d81fb1ae1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f606ee619ff4731aedd2b9d81fb1ae1" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46209 } } }
I20260929 18:42:28.855338 11428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:28.855343 11433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:28.858340 11433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:28.864298 11433 catalog_manager.cc:1347] Generated new cluster ID: f2a7e2f435c5463a988e19a6b032262b
I20260929 18:42:28.864439 11433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:28.880474 11433 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:28.881508 11433 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:28.892935 11433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f606ee619ff4731aedd2b9d81fb1ae1: Generated new TSK 0
I20260929 18:42:28.893743 11433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:42:28.929821 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:46209
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:29.040959 11448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:29.041231 11448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:29.041273 11448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:29.045320 11448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:29.045382 11448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:29.045440 11448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:42:29.050835 11448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:46209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:29.051975 11448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:29.052999 11448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:29.062925 11453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:29.062991 11448 server_base.cc:1034] running on GCE node
W20260929 18:42:29.062935 11456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:29.062951 11454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:29.063582 11448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:29.064226 11448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:29.065375 11448 hybrid_clock.cc:648] HybridClock initialized: now 1790707349065350 us; error 39 us; skew 500 ppm
Sep 29 18:42:29 dist-test-slave-smpn krb5kdc[11341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707349, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:29.068046 11448 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:42:29.068965 11448 webserver.cc:460] Webserver started at http://127.7.157.65:32869/ using document root <none> and password file <none>
I20260929 18:42:29.069504 11448 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:29.069561 11448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:29.069732 11448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:29.071806 11448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "e62268305c524b6781ad44d0a25c3358"
format_stamp: "Formatted at 2026-09-29 18:42:29 on dist-test-slave-smpn"
server_key: "7def6fd9fe525184df0a065de8b1d9a4"
server_key_iv: "d09b942d68a3eb1dfbad833f77eda835"
server_key_version: "encryptionkey@0"
I20260929 18:42:29.072307 11448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "e62268305c524b6781ad44d0a25c3358"
format_stamp: "Formatted at 2026-09-29 18:42:29 on dist-test-slave-smpn"
server_key: "7def6fd9fe525184df0a065de8b1d9a4"
server_key_iv: "d09b942d68a3eb1dfbad833f77eda835"
server_key_version: "encryptionkey@0"
I20260929 18:42:29.075923 11448 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:42:29.078496 11463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:29.079772 11448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:29.079874 11448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "e62268305c524b6781ad44d0a25c3358"
format_stamp: "Formatted at 2026-09-29 18:42:29 on dist-test-slave-smpn"
server_key: "7def6fd9fe525184df0a065de8b1d9a4"
server_key_iv: "d09b942d68a3eb1dfbad833f77eda835"
server_key_version: "encryptionkey@0"
I20260929 18:42:29.079968 11448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:29.092741 11448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:29.095594 11448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:29.095769 11448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:29.096354 11448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:29.097184 11448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:29.097229 11448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:29.097267 11448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:29.097289 11448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:29.105495 11448 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:43095
I20260929 18:42:29.105544 11576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:43095 every 8 connection(s)
I20260929 18:42:29.106388 11448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:42:29.115907 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 11448
I20260929 18:42:29.116036 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
Sep 29 18:42:29 dist-test-slave-smpn krb5kdc[11341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707349, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:29.116330 7797 external_mini_cluster.cc:1428] Setting key 57c545f3d4787baef5202c77c29bf38e
I20260929 18:42:29.120601 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:29.108182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:51867 (local address 127.7.157.126:46209)
0929 18:42:29.108353 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:29.108357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:29.109222 (+ 865us) server_negotiation.cc:407] Connection header received
0929 18:42:29.110683 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:29.110686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:29.110746 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:29.110828 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:29.112180 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:29.112772 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:29.113905 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:29.114134 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:29.114223 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:29.116825 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:29.116829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:29.116831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:29.116859 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:29.118209 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:29.118823 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:29.118826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:29.118827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:29.118915 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:29.119352 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:29.119356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:29.119358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:29.119610 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:29.119722 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:29.120272 (+ 550us) server_negotiation.cc:299] Negotiation successful
0929 18:42:29.120416 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 18:42:29.121505 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:29.108548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:46209 (local address 127.7.157.65:51867)
0929 18:42:29.109050 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:29.109096 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:29.110469 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:29.111014 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:29.111021 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:29.111476 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:29.111976 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:29.111991 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:29.112897 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:29.112901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:29.113793 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:29.113802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:29.114278 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:29.114282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:29.114450 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:29.114989 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:29.115005 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:29.116642 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:29.118321 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:29.118328 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:29.118348 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:29.118698 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:29.119004 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:29.119007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:29.119009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:29.119221 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:29.119730 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:29.119736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:29.120000 (+ 264us) client_negotiation.cc:769] Sending connection context
0929 18:42:29.120166 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 18:42:29.120442 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":124,"threads_started":1}
I20260929 18:42:29.122828 11577 heartbeater.cc:347] Connected to a master server at 127.7.157.126:46209
I20260929 18:42:29.123112 11577 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:29.123661 11577 heartbeater.cc:511] Master 127.7.157.126:46209 requested a full tablet report, sending...
I20260929 18:42:29.125264 11386 ts_manager.cc:194] Registered new tserver with Master: e62268305c524b6781ad44d0a25c3358 (127.7.157.65:43095)
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
I20260929 18:42:29.126420 11386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:51867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:42:29.141670 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:46209
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:29.248349 11582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:29.248636 11582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:29.248677 11582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:29.252738 11582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:29.252806 11582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:29.252882 11582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:42:29.258339 11582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:46209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:29.259416 11582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:29.260416 11582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:29.269598 11587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:29.269603 11590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:29.269701 11582 server_base.cc:1034] running on GCE node
W20260929 18:42:29.269603 11588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:29.270222 11582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:29.270809 11582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:29.271971 11582 hybrid_clock.cc:648] HybridClock initialized: now 1790707349271930 us; error 58 us; skew 500 ppm
Sep 29 18:42:29 dist-test-slave-smpn krb5kdc[11341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707349, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:29.274533 11582 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:42:29.275425 11582 webserver.cc:460] Webserver started at http://127.7.157.66:33709/ using document root <none> and password file <none>
I20260929 18:42:29.275923 11582 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:29.275985 11582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:29.276140 11582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:29.278267 11582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "82597594ecdb42869d3d35dc6f990450"
format_stamp: "Formatted at 2026-09-29 18:42:29 on dist-test-slave-smpn"
server_key: "d3264a91806b191363c0928557d63774"
server_key_iv: "9644df683ff69a72a35fa375e041a563"
server_key_version: "encryptionkey@0"
I20260929 18:42:29.278771 11582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "82597594ecdb42869d3d35dc6f990450"
format_stamp: "Formatted at 2026-09-29 18:42:29 on dist-test-slave-smpn"
server_key: "d3264a91806b191363c0928557d63774"
server_key_iv: "9644df683ff69a72a35fa375e041a563"
server_key_version: "encryptionkey@0"
I20260929 18:42:29.282294 11582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:42:29.284694 11597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:29.285904 11582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:29.286000 11582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "82597594ecdb42869d3d35dc6f990450"
format_stamp: "Formatted at 2026-09-29 18:42:29 on dist-test-slave-smpn"
server_key: "d3264a91806b191363c0928557d63774"
server_key_iv: "9644df683ff69a72a35fa375e041a563"
server_key_version: "encryptionkey@0"
I20260929 18:42:29.286082 11582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:29.305254 11582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:29.308332 11582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:29.308501 11582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:29.309039 11582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:29.309867 11582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:29.309914 11582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:29.309966 11582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:29.309995 11582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:29.318416 11582 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:36109
I20260929 18:42:29.318495 11710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:36109 every 8 connection(s)
I20260929 18:42:29.319376 11582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:42:29.327690 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 11582
I20260929 18:42:29.327819 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:42:29.328119 7797 external_mini_cluster.cc:1428] Setting key f90c60bbaa41333949eab8af7dfc1d5e
Sep 29 18:42:29 dist-test-slave-smpn krb5kdc[11341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707349, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:29.332969 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:29.321279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:41061 (local address 127.7.157.126:46209)
0929 18:42:29.321397 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:29.321401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:29.322196 (+ 795us) server_negotiation.cc:407] Connection header received
0929 18:42:29.323599 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:29.323603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:29.323661 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:29.323734 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:29.324920 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:29.325455 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:29.326579 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:29.326794 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:29.326882 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:29.329553 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:29.329556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:29.329559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:29.329587 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:29.330967 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:29.331462 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:29.331466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:29.331467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:29.331556 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:29.331852 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:29.331855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:29.331856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:29.332067 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:29.332153 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:29.332742 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 18:42:29.332847 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:42:29.334193 11713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:29.321587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:46209 (local address 127.7.157.66:41061)
0929 18:42:29.322034 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:29.322080 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:29.323379 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:29.323919 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:29.323927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:29.324280 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:29.324766 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:29.324779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:29.325580 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:29.325583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:29.326454 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:29.326462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:29.326935 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:29.326939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:29.327142 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:29.327721 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:29.327742 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:29.329370 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:29.331102 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:29.331108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:29.331123 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:29.331376 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:29.331638 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:29.331640 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:29.331641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:29.331768 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:29.332162 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:29.332169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:29.332491 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 18:42:29.332697 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 18:42:29.333040 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
I20260929 18:42:29.335469 11711 heartbeater.cc:347] Connected to a master server at 127.7.157.126:46209
I20260929 18:42:29.335706 11711 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:29.336174 11711 heartbeater.cc:511] Master 127.7.157.126:46209 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
I20260929 18:42:29.337177 11386 ts_manager.cc:194] Registered new tserver with Master: 82597594ecdb42869d3d35dc6f990450 (127.7.157.66:36109)
I20260929 18:42:29.337775 11386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:41061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:42:29.354441 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:46209
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:29.460903 11716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:29.461154 11716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:29.461195 11716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:29.465250 11716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:29.465312 11716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:29.465373 11716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:42:29.470712 11716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:46209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:29.471761 11716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:29.472770 11716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:42:29.482152 11716 server_base.cc:1034] running on GCE node
W20260929 18:42:29.482236 11724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:29.482205 11722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:29.482371 11721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:29.482780 11716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:29.483492 11716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:29.484642 11716 hybrid_clock.cc:648] HybridClock initialized: now 1790707349484623 us; error 34 us; skew 500 ppm
Sep 29 18:42:29 dist-test-slave-smpn krb5kdc[11341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707349, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:29.487285 11716 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:42:29.488096 11716 webserver.cc:460] Webserver started at http://127.7.157.67:43941/ using document root <none> and password file <none>
I20260929 18:42:29.488621 11716 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:29.488683 11716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:29.488873 11716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:29.491253 11716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "5915ed22b8f04368bcd69c484c97171d"
format_stamp: "Formatted at 2026-09-29 18:42:29 on dist-test-slave-smpn"
server_key: "09bf2a40b6b35a9c949c1d892a4978aa"
server_key_iv: "3b5d2da0400bb1c1ca01c2c802b69954"
server_key_version: "encryptionkey@0"
I20260929 18:42:29.491786 11716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "5915ed22b8f04368bcd69c484c97171d"
format_stamp: "Formatted at 2026-09-29 18:42:29 on dist-test-slave-smpn"
server_key: "09bf2a40b6b35a9c949c1d892a4978aa"
server_key_iv: "3b5d2da0400bb1c1ca01c2c802b69954"
server_key_version: "encryptionkey@0"
I20260929 18:42:29.495342 11716 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:42:29.497712 11731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:29.498929 11716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:29.499028 11716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "5915ed22b8f04368bcd69c484c97171d"
format_stamp: "Formatted at 2026-09-29 18:42:29 on dist-test-slave-smpn"
server_key: "09bf2a40b6b35a9c949c1d892a4978aa"
server_key_iv: "3b5d2da0400bb1c1ca01c2c802b69954"
server_key_version: "encryptionkey@0"
I20260929 18:42:29.499159 11716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:29.510151 11716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:29.513206 11716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:29.513360 11716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:29.513892 11716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:29.514686 11716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:29.514730 11716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:29.514767 11716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:29.514787 11716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:29.522819 11716 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:35531
I20260929 18:42:29.522873 11844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:35531 every 8 connection(s)
I20260929 18:42:29.523752 11716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:42:29.529950 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 11716
I20260929 18:42:29.530084 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:42:29.530369 7797 external_mini_cluster.cc:1428] Setting key 2395006a9c9970b6beb637a300635280
Sep 29 18:42:29 dist-test-slave-smpn krb5kdc[11341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707349, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:29.537110 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:29.525602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:36549 (local address 127.7.157.126:46209)
0929 18:42:29.525749 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:29.525753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:29.526602 (+ 849us) server_negotiation.cc:407] Connection header received
0929 18:42:29.527998 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:29.528001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:29.528060 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:29.528136 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:29.529339 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:29.529987 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:29.531397 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:29.531646 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:29.531756 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:29.534096 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:29.534099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:29.534101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:29.534125 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:29.535267 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:29.535699 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:29.535701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:29.535702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:29.535769 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:29.536056 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:29.536059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:29.536060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:29.536300 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:29.536401 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:29.536860 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 18:42:29.536977 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:42:29.537863 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:29.525990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:46209 (local address 127.7.157.67:36549)
0929 18:42:29.526450 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:29.526494 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:29.527787 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:29.528332 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:29.528339 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:29.528704 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:29.529194 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:29.529207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:29.530158 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:29.530162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:29.531293 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:29.531303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:29.531800 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:29.531803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:29.531954 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:29.532542 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:29.532559 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:29.533950 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:29.535355 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:29.535360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:29.535374 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:29.535615 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:29.535829 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:29.535831 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:29.535833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:29.535955 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:29.536392 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:29.536397 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:29.536629 (+ 232us) client_negotiation.cc:769] Sending connection context
0929 18:42:29.536800 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 18:42:29.537050 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260929 18:42:29.538897 11845 heartbeater.cc:347] Connected to a master server at 127.7.157.126:46209
I20260929 18:42:29.539131 11845 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:29.539592 11845 heartbeater.cc:511] Master 127.7.157.126:46209 requested a full tablet report, sending...
I20260929 18:42:29.540537 11386 ts_manager.cc:194] Registered new tserver with Master: 5915ed22b8f04368bcd69c484c97171d (127.7.157.67:35531)
I20260929 18:42:29.541055 11386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:36549
I20260929 18:42:29.543934 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:42:29 dist-test-slave-smpn krb5kdc[11341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:42:29 dist-test-slave-smpn krb5kdc[11341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.65@KRBTEST.COM
I20260929 18:42:29.568172 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:29.559138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.7.157.65:43095)
0929 18:42:29.559454 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:29.559467 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:29.559493 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:42:29.559542 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:29.559547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:29.559756 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:29.559885 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:29.560569 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:29.561336 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:29.562372 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:29.562772 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:29.562875 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:29.565103 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:29.565111 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:29.565125 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:29.565150 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:29.566459 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:29.566852 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:29.566858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:29.566859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:29.566936 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:29.567223 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:29.567227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:29.567229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:29.567469 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:29.567672 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:29.567856 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 18:42:29.568007 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
Sep 29 18:42:29 dist-test-slave-smpn krb5kdc[11341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:29.578016 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:29.570899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.7.157.126:46209)
0929 18:42:29.571042 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:29.571046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:29.571061 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:42:29.571207 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:29.571209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:29.571249 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:29.571310 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:29.571815 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:29.572308 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:29.573000 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:29.573194 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:29.573259 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:29.575418 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:29.575421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:29.575422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:29.575447 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:29.576524 (+ 1077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:29.576861 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:29.576863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:29.576865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:29.576931 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:29.577163 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:29.577168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:29.577170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:29.577395 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:29.577519 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:29.577744 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 18:42:29.577859 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 18:42:29.578402 11386 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40120
I20260929 18:42:29.578933 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 11448
I20260929 18:42:29.583602 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 11582
I20260929 18:42:29.588093 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 11716
I20260929 18:42:29.592331 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 11354
2026-09-29T18:42:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2008 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:42:29 dist-test-slave-smpn krb5kdc[11857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:42:29 dist-test-slave-smpn krb5kdc[11857](info): set up 2 sockets
Sep 29 18:42:29 dist-test-slave-smpn krb5kdc[11857](info): commencing operation
krb5kdc: starting...
W20260929 18:42:30.550980 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:42:30 dist-test-slave-smpn krb5kdc[11857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:42:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:42:30.666220 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:43347
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:43347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:30.777297 11870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:30.777635 11870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:30.777701 11870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:30.781988 11870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:30.782070 11870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:30.782097 11870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:30.782119 11870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:30.782140 11870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:30.787761 11870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:43347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:43347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:30.788964 11870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:30.790215 11870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:30.799330 11876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:30.799350 11875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:30.799504 11870 server_base.cc:1034] running on GCE node
W20260929 18:42:30.799330 11878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:30.800405 11870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:30.801788 11870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:30.802980 11870 hybrid_clock.cc:648] HybridClock initialized: now 1790707350802957 us; error 40 us; skew 500 ppm
Sep 29 18:42:30 dist-test-slave-smpn krb5kdc[11857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707350, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:30.808861 11870 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:42:30.809655 11870 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:30.809733 11870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:30.809914 11870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:30.812160 11870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "f1136f3e221d4241bc00cd89c78d825c"
format_stamp: "Formatted at 2026-09-29 18:42:30 on dist-test-slave-smpn"
server_key: "d3fbbf7e90a1d7374ebb8fa996206d4e"
server_key_iv: "f0f84b075b6ede04ea60d5bd99414731"
server_key_version: "encryptionkey@0"
I20260929 18:42:30.812724 11870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "f1136f3e221d4241bc00cd89c78d825c"
format_stamp: "Formatted at 2026-09-29 18:42:30 on dist-test-slave-smpn"
server_key: "d3fbbf7e90a1d7374ebb8fa996206d4e"
server_key_iv: "f0f84b075b6ede04ea60d5bd99414731"
server_key_version: "encryptionkey@0"
I20260929 18:42:30.816624 11870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:42:30.819296 11884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:30.820642 11870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:42:30.820779 11870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "f1136f3e221d4241bc00cd89c78d825c"
format_stamp: "Formatted at 2026-09-29 18:42:30 on dist-test-slave-smpn"
server_key: "d3fbbf7e90a1d7374ebb8fa996206d4e"
server_key_iv: "f0f84b075b6ede04ea60d5bd99414731"
server_key_version: "encryptionkey@0"
I20260929 18:42:30.820896 11870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:30.852151 11870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:30.855194 11870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:30.855443 11870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:30.862317 11870 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:43347
I20260929 18:42:30.862555 11936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:43347 every 8 connection(s)
I20260929 18:42:30.863019 11870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:30.865796 11937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:30.871569 11937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c: Bootstrap starting.
I20260929 18:42:30.872483 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 11870
I20260929 18:42:30.872623 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:30.872925 7797 external_mini_cluster.cc:1428] Setting key f9d19554ba8bfd1d6491a583bc0a4764
I20260929 18:42:30.874585 11937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:30.875357 11937 log.cc:824] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:30.877213 11937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c: No bootstrap required, opened a new log
I20260929 18:42:30.880744 11937 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1136f3e221d4241bc00cd89c78d825c" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 43347 } }
I20260929 18:42:30.881042 11937 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:30.881102 11937 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1136f3e221d4241bc00cd89c78d825c, State: Initialized, Role: FOLLOWER
Sep 29 18:42:30 dist-test-slave-smpn krb5kdc[11857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:30.881584 11937 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1136f3e221d4241bc00cd89c78d825c" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 43347 } }
I20260929 18:42:30.881723 11937 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:30.881807 11937 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:30.881907 11937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:30.882974 11937 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1136f3e221d4241bc00cd89c78d825c" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 43347 } }
I20260929 18:42:30.883328 11937 leader_election.cc:302] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1136f3e221d4241bc00cd89c78d825c; no voters:
I20260929 18:42:30.883620 11937 leader_election.cc:288] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:30.883733 11942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:30.883970 11942 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [term 1 LEADER]: Becoming Leader. State: Replica: f1136f3e221d4241bc00cd89c78d825c, State: Running, Role: LEADER
I20260929 18:42:30.884334 11942 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1136f3e221d4241bc00cd89c78d825c" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 43347 } }
I20260929 18:42:30.884975 11937 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:30.886039 11943 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1136f3e221d4241bc00cd89c78d825c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1136f3e221d4241bc00cd89c78d825c" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 43347 } } }
I20260929 18:42:30.886161 11943 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:30.886533 11944 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1136f3e221d4241bc00cd89c78d825c. Latest consensus state: current_term: 1 leader_uuid: "f1136f3e221d4241bc00cd89c78d825c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1136f3e221d4241bc00cd89c78d825c" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 43347 } } }
I20260929 18:42:30.886701 11944 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:30.887544 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:30.874076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.7.157.126:43347)
0929 18:42:30.874508 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:30.874525 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:30.874561 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:42:30.875294 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:30.875313 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:30.875690 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:30.876000 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:30.876766 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:30.877780 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:30.878621 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:30.879166 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:30.879315 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:30.881838 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:30.881843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:30.881858 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:30.881888 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:30.883629 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:30.884196 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:30.884201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:30.884208 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:30.884345 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:30.884674 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:30.884678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:30.884679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:30.885205 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:30.885397 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:30.886134 (+ 737us) server_negotiation.cc:299] Negotiation successful
0929 18:42:30.886449 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":127,"threads_started":1}
W20260929 18:42:30.890200 11957 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:42:30.890280 11957 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:42:30.890336 11951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:30.891232 11951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:30.896806 11951 catalog_manager.cc:1347] Generated new cluster ID: ec5b548871cf40849ec168e520851a9e
I20260929 18:42:30.896881 11951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:30.906687 11951 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:30.908001 11951 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:30.919839 11951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c: Generated new TSK 0
I20260929 18:42:30.920629 11951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:42:30.954622 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:43347
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:31.091289 11963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:31.091591 11963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:31.091638 11963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:31.095763 11963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:31.095832 11963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:31.095896 11963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:42:31.101373 11963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:43347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:31.102465 11963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:31.103529 11963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:31.113117 11968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:31.113089 11971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:31.113235 11963 server_base.cc:1034] running on GCE node
W20260929 18:42:31.113089 11969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:31.113790 11963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:31.114454 11963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:31.115607 11963 hybrid_clock.cc:648] HybridClock initialized: now 1790707351115587 us; error 31 us; skew 500 ppm
Sep 29 18:42:31 dist-test-slave-smpn krb5kdc[11857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707351, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:31.118314 11963 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:42:31.119051 11963 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:31.119160 11963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:31.119377 11963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:31.121398 11963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "0ec7637d1cd643f79d025ba05f198dc6"
format_stamp: "Formatted at 2026-09-29 18:42:31 on dist-test-slave-smpn"
server_key: "e9e85f5ca6088fd4b71a0829b519b6a6"
server_key_iv: "c37f67a9bd7692380e4c0e46a8092064"
server_key_version: "encryptionkey@0"
I20260929 18:42:31.121930 11963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "0ec7637d1cd643f79d025ba05f198dc6"
format_stamp: "Formatted at 2026-09-29 18:42:31 on dist-test-slave-smpn"
server_key: "e9e85f5ca6088fd4b71a0829b519b6a6"
server_key_iv: "c37f67a9bd7692380e4c0e46a8092064"
server_key_version: "encryptionkey@0"
I20260929 18:42:31.125689 11963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 18:42:31.128121 11977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:31.129458 11963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:31.129566 11963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "0ec7637d1cd643f79d025ba05f198dc6"
format_stamp: "Formatted at 2026-09-29 18:42:31 on dist-test-slave-smpn"
server_key: "e9e85f5ca6088fd4b71a0829b519b6a6"
server_key_iv: "c37f67a9bd7692380e4c0e46a8092064"
server_key_version: "encryptionkey@0"
I20260929 18:42:31.129662 11963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:31.158483 11963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:31.161428 11963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:31.161604 11963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:31.162196 11963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:31.163012 11963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:31.163057 11963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:31.163136 11963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:31.163164 11963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:31.171693 11963 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:40245
I20260929 18:42:31.171716 12090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:40245 every 8 connection(s)
I20260929 18:42:31.172274 11963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:42:31 dist-test-slave-smpn krb5kdc[11857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707351, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:31.182066 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 11963
I20260929 18:42:31.182180 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:42:31.182444 7797 external_mini_cluster.cc:1428] Setting key c3c275768c22a5fe9d3022039f339c8c
I20260929 18:42:31.184806 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.173914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:48191 (local address 127.7.157.126:43347)
0929 18:42:31.174076 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:31.174081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:31.174919 (+ 838us) server_negotiation.cc:407] Connection header received
0929 18:42:31.176141 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:31.176144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:31.176191 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:31.176260 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:31.177345 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.177867 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.178880 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.179133 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.179202 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.181434 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:31.181437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:31.181439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:31.181463 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:31.182789 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:31.183319 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:31.183321 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:31.183323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:31.183397 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:31.183774 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:31.183777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:31.183779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:31.183983 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:31.184076 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:31.184519 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 18:42:31.184654 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:42:31.185534 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.174326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:43347 (local address 127.7.157.65:48191)
0929 18:42:31.174766 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:31.174808 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:31.175943 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:31.176412 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:31.176419 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:31.176754 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:31.177236 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:31.177250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.177951 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.177953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:31.178797 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:31.178805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.179249 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.179252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:31.179384 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.179932 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:31.179947 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:31.181274 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:31.182891 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:31.182897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:31.182911 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:31.183226 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:31.183473 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:31.183475 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:31.183478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:31.183681 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:31.184075 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:31.184080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:31.184302 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 18:42:31.184451 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 18:42:31.184694 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260929 18:42:31.186694 12091 heartbeater.cc:347] Connected to a master server at 127.7.157.126:43347
I20260929 18:42:31.186909 12091 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:31.187392 12091 heartbeater.cc:511] Master 127.7.157.126:43347 requested a full tablet report, sending...
I20260929 18:42:31.188737 11901 ts_manager.cc:194] Registered new tserver with Master: 0ec7637d1cd643f79d025ba05f198dc6 (127.7.157.65:40245)
I20260929 18:42:31.189823 11901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:48191
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:42:31.206607 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:43347
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:31.313660 12096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:31.313901 12096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:31.313953 12096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:31.317957 12096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:31.318020 12096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:31.318094 12096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:42:31.323228 12096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:43347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:31.324178 12096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:31.325136 12096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:31.334079 12104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:31.334079 12102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:31.334079 12101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:31.334494 12096 server_base.cc:1034] running on GCE node
I20260929 18:42:31.334904 12096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:31.335577 12096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:31.336727 12096 hybrid_clock.cc:648] HybridClock initialized: now 1790707351336707 us; error 30 us; skew 500 ppm
Sep 29 18:42:31 dist-test-slave-smpn krb5kdc[11857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707351, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:31.339368 12096 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:42:31.340041 12096 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:31.340107 12096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:31.340296 12096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:31.342332 12096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "0c3eb0a5a11d4013ad3576702e194fbd"
format_stamp: "Formatted at 2026-09-29 18:42:31 on dist-test-slave-smpn"
server_key: "8b19dbd22ebcd6ea9d91dd7e6dd3dcd8"
server_key_iv: "6737c931b5b52985aa0d7d9b0defb211"
server_key_version: "encryptionkey@0"
I20260929 18:42:31.342862 12096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "0c3eb0a5a11d4013ad3576702e194fbd"
format_stamp: "Formatted at 2026-09-29 18:42:31 on dist-test-slave-smpn"
server_key: "8b19dbd22ebcd6ea9d91dd7e6dd3dcd8"
server_key_iv: "6737c931b5b52985aa0d7d9b0defb211"
server_key_version: "encryptionkey@0"
I20260929 18:42:31.346515 12096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:42:31.348874 12110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:31.350044 12096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:31.350153 12096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "0c3eb0a5a11d4013ad3576702e194fbd"
format_stamp: "Formatted at 2026-09-29 18:42:31 on dist-test-slave-smpn"
server_key: "8b19dbd22ebcd6ea9d91dd7e6dd3dcd8"
server_key_iv: "6737c931b5b52985aa0d7d9b0defb211"
server_key_version: "encryptionkey@0"
I20260929 18:42:31.350265 12096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:31.365312 12096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:31.368674 12096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:31.368853 12096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:31.369448 12096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:31.370280 12096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:31.370326 12096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:31.370378 12096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:31.370406 12096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:31.378661 12096 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:39329
I20260929 18:42:31.378700 12223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:39329 every 8 connection(s)
I20260929 18:42:31.379268 12096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:42:31.382359 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 12096
I20260929 18:42:31.382467 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:42:31.382727 7797 external_mini_cluster.cc:1428] Setting key a133f1f80496fcc0b7bbf75447f9f6f2
Sep 29 18:42:31 dist-test-slave-smpn krb5kdc[11857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707351, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
I20260929 18:42:31.392871 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.380864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:37687 (local address 127.7.157.126:43347)
0929 18:42:31.380985 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:31.380989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:31.382136 (+ 1147us) server_negotiation.cc:407] Connection header received
0929 18:42:31.383103 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:31.383106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:31.383161 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:31.383254 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:31.384595 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.385309 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.386899 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.387152 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.387257 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.389598 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:31.389601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:31.389603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:31.389628 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:31.390883 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:31.391392 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:31.391396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:31.391398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:31.391483 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:31.391783 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:31.391786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:31.391788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:31.391961 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:31.392046 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:31.392574 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 18:42:31.392707 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
I20260929 18:42:31.393784 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.381247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:43347 (local address 127.7.157.66:37687)
0929 18:42:31.381988 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:31.382029 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:31.382888 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:31.383438 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:31.383447 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:31.383903 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:31.384456 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:31.384470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.385425 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.385428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:31.386787 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:31.386798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.387292 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.387296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:31.387491 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.388019 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:31.388037 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:31.389442 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:31.390986 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:31.390992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:31.391009 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:31.391303 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:31.391561 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:31.391563 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:31.391566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:31.391700 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:31.392061 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:31.392068 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:31.392353 (+ 285us) client_negotiation.cc:769] Sending connection context
0929 18:42:31.392519 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 18:42:31.392844 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260929 18:42:31.394860 12224 heartbeater.cc:347] Connected to a master server at 127.7.157.126:43347
I20260929 18:42:31.395125 12224 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:31.395565 12224 heartbeater.cc:511] Master 127.7.157.126:43347 requested a full tablet report, sending...
I20260929 18:42:31.396560 11901 ts_manager.cc:194] Registered new tserver with Master: 0c3eb0a5a11d4013ad3576702e194fbd (127.7.157.66:39329)
I20260929 18:42:31.397275 11901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:37687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:42:31.408164 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:43347
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:31.517861 12229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:31.518141 12229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:31.518194 12229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:31.522584 12229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:31.522660 12229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:31.522727 12229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:42:31.528154 12229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:43347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:31.529196 12229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:31.530191 12229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:42:31.539981 12229 server_base.cc:1034] running on GCE node
W20260929 18:42:31.539950 12237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:31.539942 12234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:31.539916 12235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:31.540551 12229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:31.541205 12229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:31.542352 12229 hybrid_clock.cc:648] HybridClock initialized: now 1790707351542328 us; error 32 us; skew 500 ppm
Sep 29 18:42:31 dist-test-slave-smpn krb5kdc[11857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707351, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:31.545090 12229 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:42:31.545810 12229 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:31.545878 12229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:31.546065 12229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:31.548200 12229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "65c3629afeb143c891d1118564abac20"
format_stamp: "Formatted at 2026-09-29 18:42:31 on dist-test-slave-smpn"
server_key: "bd9ff926c895bb8ebda147f3c34e558d"
server_key_iv: "6de4c03cd3f0173670af23c98bb417ca"
server_key_version: "encryptionkey@0"
I20260929 18:42:31.548841 12229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "65c3629afeb143c891d1118564abac20"
format_stamp: "Formatted at 2026-09-29 18:42:31 on dist-test-slave-smpn"
server_key: "bd9ff926c895bb8ebda147f3c34e558d"
server_key_iv: "6de4c03cd3f0173670af23c98bb417ca"
server_key_version: "encryptionkey@0"
I20260929 18:42:31.552758 12229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:42:31.555282 12243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:31.556481 12229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:31.556595 12229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "65c3629afeb143c891d1118564abac20"
format_stamp: "Formatted at 2026-09-29 18:42:31 on dist-test-slave-smpn"
server_key: "bd9ff926c895bb8ebda147f3c34e558d"
server_key_iv: "6de4c03cd3f0173670af23c98bb417ca"
server_key_version: "encryptionkey@0"
I20260929 18:42:31.556689 12229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:31.579543 12229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:31.582671 12229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:31.582846 12229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:31.583544 12229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:31.584463 12229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:31.584510 12229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:31.584563 12229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:31.584591 12229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:31.593111 12229 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:45273
I20260929 18:42:31.593132 12356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:45273 every 8 connection(s)
I20260929 18:42:31.593762 12229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:42:31.594177 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 12229
I20260929 18:42:31.594296 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:42:31.594563 7797 external_mini_cluster.cc:1428] Setting key 97b5d30ce2bf91a4978b6dd9e9647fa7
Sep 29 18:42:31 dist-test-slave-smpn krb5kdc[11857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707351, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:31.606802 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.595580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:56533 (local address 127.7.157.126:43347)
0929 18:42:31.595725 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:31.595730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:31.596854 (+ 1124us) server_negotiation.cc:407] Connection header received
0929 18:42:31.598080 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:31.598082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:31.598126 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:31.598195 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:31.599386 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.599906 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.600982 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.601186 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.601251 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.603686 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:31.603689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:31.603691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:31.603714 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:31.604898 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:31.605361 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:31.605365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:31.605367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:31.605446 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:31.605754 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:31.605757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:31.605759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:31.605942 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:31.606025 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:31.606508 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 18:42:31.606629 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:42:31.607712 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.596173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:43347 (local address 127.7.157.67:56533)
0929 18:42:31.596724 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:31.596772 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:31.597893 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:31.598349 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:31.598355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:31.598716 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:31.599267 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:31.599283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.599995 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.599998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:31.600892 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:31.600901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.601291 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.601294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:31.601436 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.601974 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:31.601990 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:31.603524 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:31.605002 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:31.605008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:31.605024 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:31.605269 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:31.605524 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:31.605527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:31.605529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:31.605671 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:31.606036 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:31.606044 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:31.606286 (+ 242us) client_negotiation.cc:769] Sending connection context
0929 18:42:31.606453 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 18:42:31.606794 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":140,"threads_started":1}
I20260929 18:42:31.608840 12357 heartbeater.cc:347] Connected to a master server at 127.7.157.126:43347
I20260929 18:42:31.609074 12357 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:31.609519 12357 heartbeater.cc:511] Master 127.7.157.126:43347 requested a full tablet report, sending...
I20260929 18:42:31.610579 11901 ts_manager.cc:194] Registered new tserver with Master: 65c3629afeb143c891d1118564abac20 (127.7.157.67:45273)
I20260929 18:42:31.611186 11901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:56533
I20260929 18:42:31.620530 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:42:31.629518 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.622414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.7.157.126:43347)
0929 18:42:31.622559 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:31.622564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:31.622726 (+ 162us) server_negotiation.cc:407] Connection header received
0929 18:42:31.622848 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:31.622851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:31.622898 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:31.622961 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:31.623691 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.624277 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.625161 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.625425 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.625514 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.626777 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:31.626781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:31.626784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:31.626812 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:31.627982 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:31.628359 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:31.628363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:31.628365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:31.628441 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:31.628673 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:31.628677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:31.628679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:31.628906 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:31.629025 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:31.629247 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 18:42:31.629360 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:42:31.632411 11901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:42:31.635660 11901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:31.649806 12369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.644107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:45273 (local address 127.0.0.1:52466)
0929 18:42:31.644548 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:31.644597 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:31.644720 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:31.645572 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:31.645576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:31.645597 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:31.645710 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:31.645715 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.647113 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.647117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:31.648313 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:31.648323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.649296 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.649299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:31.649404 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.649446 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 18:42:31.649543 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 18:42:31.649665 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":85,"threads_started":1}
I20260929 18:42:31.649942 12373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.645291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.7.157.66:39329)
0929 18:42:31.645666 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:31.645690 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:31.645710 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:42:31.645778 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:31.645783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:31.645964 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:31.646098 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:31.646347 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.647211 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.648665 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.649423 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.649519 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.649650 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:31.649714 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 18:42:31.649797 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":99,"threads_started":1}
I20260929 18:42:31.650380 12370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.644216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.7.157.67:45273)
0929 18:42:31.645004 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:31.645017 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:31.645039 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:42:31.645114 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:31.645119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:31.645341 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:31.645538 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:31.645822 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.646975 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.648423 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.649168 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.649285 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.649431 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:31.650190 (+ 759us) server_negotiation.cc:299] Negotiation successful
0929 18:42:31.650270 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":90,"threads_started":1}
I20260929 18:42:31.650414 12372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.644992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.7.157.65:40245)
0929 18:42:31.645445 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:31.645451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:31.645885 (+ 434us) server_negotiation.cc:407] Connection header received
0929 18:42:31.645984 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:31.645989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:31.646177 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:31.646317 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:31.646498 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.647498 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.648835 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.649421 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.649524 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.649667 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:31.650195 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 18:42:31.650295 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":89,"threads_started":1}
I20260929 18:42:31.649950 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.644189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:39329 (local address 127.0.0.1:56694)
0929 18:42:31.644766 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:31.644779 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:31.644857 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:31.646090 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:31.646092 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:31.646108 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:31.646225 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:31.646233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.647339 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.647342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:31.648541 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:31.648550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.649520 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.649523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:31.649608 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.649629 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:42:31.649687 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:42:31.649753 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260929 18:42:31.650215 12371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.644801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:40245 (local address 127.0.0.1:55826)
0929 18:42:31.645781 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:31.645794 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:31.645889 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:31.646306 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:31.646308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:31.646327 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:31.646416 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:31.646422 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.647657 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.647661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:31.648736 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:31.648743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.649854 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.649857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:31.649941 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.649961 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:42:31.650025 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:42:31.650089 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":85,"threads_started":1}
I20260929 18:42:31.653801 12158 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3c2fc5acfd48c696d189694c2a2351 (DEFAULT_TABLE table=test-table [id=b5f4b3bf414948ee82ed4d9473b7f45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:31.654204 12025 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3c2fc5acfd48c696d189694c2a2351 (DEFAULT_TABLE table=test-table [id=b5f4b3bf414948ee82ed4d9473b7f45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:31.654986 12158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d3c2fc5acfd48c696d189694c2a2351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:31.655212 12025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d3c2fc5acfd48c696d189694c2a2351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:31.656103 12291 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3c2fc5acfd48c696d189694c2a2351 (DEFAULT_TABLE table=test-table [id=b5f4b3bf414948ee82ed4d9473b7f45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:31.656935 12291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d3c2fc5acfd48c696d189694c2a2351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:31.662352 12374 tablet_bootstrap.cc:492] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6: Bootstrap starting.
I20260929 18:42:31.662777 12375 tablet_bootstrap.cc:492] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd: Bootstrap starting.
I20260929 18:42:31.663348 12376 tablet_bootstrap.cc:492] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20: Bootstrap starting.
I20260929 18:42:31.665541 12376 tablet_bootstrap.cc:654] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:31.665902 12375 tablet_bootstrap.cc:654] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:31.666816 12376 log.cc:824] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:31.666816 12375 log.cc:824] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:31.668797 12374 tablet_bootstrap.cc:654] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:31.668955 12375 tablet_bootstrap.cc:492] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd: No bootstrap required, opened a new log
I20260929 18:42:31.668955 12376 tablet_bootstrap.cc:492] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20: No bootstrap required, opened a new log
I20260929 18:42:31.669153 12375 ts_tablet_manager.cc:1397] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 18:42:31.669153 12376 ts_tablet_manager.cc:1397] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:42:31.669931 12374 log.cc:824] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:31.672497 12374 tablet_bootstrap.cc:492] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6: No bootstrap required, opened a new log
I20260929 18:42:31.672703 12374 ts_tablet_manager.cc:1397] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260929 18:42:31.673039 12376 raft_consensus.cc:348] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c3629afeb143c891d1118564abac20" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 45273 } } peers { permanent_uuid: "0c3eb0a5a11d4013ad3576702e194fbd" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39329 } } peers { permanent_uuid: "0ec7637d1cd643f79d025ba05f198dc6" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40245 } }
I20260929 18:42:31.673039 12375 raft_consensus.cc:348] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c3629afeb143c891d1118564abac20" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 45273 } } peers { permanent_uuid: "0c3eb0a5a11d4013ad3576702e194fbd" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39329 } } peers { permanent_uuid: "0ec7637d1cd643f79d025ba05f198dc6" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40245 } }
I20260929 18:42:31.673332 12375 raft_consensus.cc:374] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:31.673333 12376 raft_consensus.cc:374] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:31.673389 12376 raft_consensus.cc:729] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c3629afeb143c891d1118564abac20, State: Initialized, Role: FOLLOWER
I20260929 18:42:31.673389 12375 raft_consensus.cc:729] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3eb0a5a11d4013ad3576702e194fbd, State: Initialized, Role: FOLLOWER
I20260929 18:42:31.673862 12376 consensus_queue.cc:260] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c3629afeb143c891d1118564abac20" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 45273 } } peers { permanent_uuid: "0c3eb0a5a11d4013ad3576702e194fbd" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39329 } } peers { permanent_uuid: "0ec7637d1cd643f79d025ba05f198dc6" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40245 } }
I20260929 18:42:31.673862 12375 consensus_queue.cc:260] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c3629afeb143c891d1118564abac20" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 45273 } } peers { permanent_uuid: "0c3eb0a5a11d4013ad3576702e194fbd" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39329 } } peers { permanent_uuid: "0ec7637d1cd643f79d025ba05f198dc6" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40245 } }
I20260929 18:42:31.674749 12224 heartbeater.cc:503] Master 127.7.157.126:43347 was elected leader, sending a full tablet report...
I20260929 18:42:31.674805 12357 heartbeater.cc:503] Master 127.7.157.126:43347 was elected leader, sending a full tablet report...
I20260929 18:42:31.674899 12375 ts_tablet_manager.cc:1428] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:42:31.675473 12376 ts_tablet_manager.cc:1428] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:42:31.676291 12374 raft_consensus.cc:348] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c3629afeb143c891d1118564abac20" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 45273 } } peers { permanent_uuid: "0c3eb0a5a11d4013ad3576702e194fbd" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39329 } } peers { permanent_uuid: "0ec7637d1cd643f79d025ba05f198dc6" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40245 } }
I20260929 18:42:31.676568 12374 raft_consensus.cc:374] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:31.676625 12374 raft_consensus.cc:729] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec7637d1cd643f79d025ba05f198dc6, State: Initialized, Role: FOLLOWER
I20260929 18:42:31.677064 12374 consensus_queue.cc:260] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c3629afeb143c891d1118564abac20" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 45273 } } peers { permanent_uuid: "0c3eb0a5a11d4013ad3576702e194fbd" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39329 } } peers { permanent_uuid: "0ec7637d1cd643f79d025ba05f198dc6" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40245 } }
I20260929 18:42:31.677766 12091 heartbeater.cc:503] Master 127.7.157.126:43347 was elected leader, sending a full tablet report...
I20260929 18:42:31.678159 12374 ts_tablet_manager.cc:1428] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
W20260929 18:42:31.844733 12358 tablet.cc:2369] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:42:31.880270 12225 tablet.cc:2369] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:42:31.923485 12092 tablet.cc:2369] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:42:31.959669 12380 raft_consensus.cc:482] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:31.959910 12380 raft_consensus.cc:504] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c3629afeb143c891d1118564abac20" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 45273 } } peers { permanent_uuid: "0c3eb0a5a11d4013ad3576702e194fbd" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39329 } } peers { permanent_uuid: "0ec7637d1cd643f79d025ba05f198dc6" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40245 } }
I20260929 18:42:31.961179 12380 leader_election.cc:288] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c3eb0a5a11d4013ad3576702e194fbd (127.7.157.66:39329), 0ec7637d1cd643f79d025ba05f198dc6 (127.7.157.65:40245)
I20260929 18:42:31.964696 12373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.961521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:36285 (local address 127.7.157.66:39329)
0929 18:42:31.961653 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:31.961658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:31.961675 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:42:31.961857 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:31.961862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:31.961917 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:31.961998 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:31.962252 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.962759 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.963816 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.964197 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.964302 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.964355 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:31.964512 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 18:42:31.964577 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:42:31.964697 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.961436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:39329 (local address 127.7.157.67:36285)
0929 18:42:31.961561 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:31.961577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:31.961722 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:31.962008 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:31.962011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:31.962029 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:31.962133 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:31.962139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.962861 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.962864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:31.963694 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:31.963703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.964302 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.964306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:31.964406 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.964424 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:42:31.964487 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:42:31.964559 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 18:42:31.964959 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.961706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:40245 (local address 127.7.157.67:44909)
0929 18:42:31.962040 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:31.962052 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:31.962143 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:31.962358 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:31.962361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:31.962374 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:31.962488 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:31.962493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.963179 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.963183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:31.963950 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:31.963958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.964566 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.964569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:31.964669 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.964692 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:42:31.964753 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 18:42:31.964823 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":116,"threads_started":1}
I20260929 18:42:31.965240 12372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:31.961864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:44909 (local address 127.7.157.65:40245)
0929 18:42:31.961988 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:31.961992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:31.962130 (+ 138us) server_negotiation.cc:407] Connection header received
0929 18:42:31.962236 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:31.962239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:31.962280 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:31.962341 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:31.962585 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.963060 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.964069 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:31.964455 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:31.964561 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:31.964607 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:31.965079 (+ 472us) server_negotiation.cc:299] Negotiation successful
0929 18:42:31.965156 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:42:31.965353 12178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3c2fc5acfd48c696d189694c2a2351" candidate_uuid: "65c3629afeb143c891d1118564abac20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c3eb0a5a11d4013ad3576702e194fbd" is_pre_election: true
I20260929 18:42:31.965651 12178 raft_consensus.cc:2454] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c3629afeb143c891d1118564abac20 in term 0.
I20260929 18:42:31.965708 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3c2fc5acfd48c696d189694c2a2351" candidate_uuid: "65c3629afeb143c891d1118564abac20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec7637d1cd643f79d025ba05f198dc6" is_pre_election: true
I20260929 18:42:31.965939 12045 raft_consensus.cc:2454] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c3629afeb143c891d1118564abac20 in term 0.
I20260929 18:42:31.966135 12245 leader_election.cc:302] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c3eb0a5a11d4013ad3576702e194fbd, 65c3629afeb143c891d1118564abac20; no voters:
I20260929 18:42:31.966411 12380 raft_consensus.cc:2781] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:31.966486 12380 raft_consensus.cc:482] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:31.966516 12380 raft_consensus.cc:3037] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:31.967551 12380 raft_consensus.cc:504] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c3629afeb143c891d1118564abac20" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 45273 } } peers { permanent_uuid: "0c3eb0a5a11d4013ad3576702e194fbd" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39329 } } peers { permanent_uuid: "0ec7637d1cd643f79d025ba05f198dc6" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40245 } }
I20260929 18:42:31.967870 12380 leader_election.cc:288] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [CANDIDATE]: Term 1 election: Requested vote from peers 0c3eb0a5a11d4013ad3576702e194fbd (127.7.157.66:39329), 0ec7637d1cd643f79d025ba05f198dc6 (127.7.157.65:40245)
I20260929 18:42:31.968248 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3c2fc5acfd48c696d189694c2a2351" candidate_uuid: "65c3629afeb143c891d1118564abac20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec7637d1cd643f79d025ba05f198dc6"
I20260929 18:42:31.968225 12178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3c2fc5acfd48c696d189694c2a2351" candidate_uuid: "65c3629afeb143c891d1118564abac20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c3eb0a5a11d4013ad3576702e194fbd"
I20260929 18:42:31.968358 12178 raft_consensus.cc:3037] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:31.968358 12045 raft_consensus.cc:3037] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:31.969553 12045 raft_consensus.cc:2454] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c3629afeb143c891d1118564abac20 in term 1.
I20260929 18:42:31.969553 12178 raft_consensus.cc:2454] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c3629afeb143c891d1118564abac20 in term 1.
I20260929 18:42:31.969834 12245 leader_election.cc:302] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec7637d1cd643f79d025ba05f198dc6, 65c3629afeb143c891d1118564abac20; no voters:
I20260929 18:42:31.969976 12380 raft_consensus.cc:2781] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:31.970184 12380 raft_consensus.cc:686] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [term 1 LEADER]: Becoming Leader. State: Replica: 65c3629afeb143c891d1118564abac20, State: Running, Role: LEADER
I20260929 18:42:31.970506 12380 consensus_queue.cc:237] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c3629afeb143c891d1118564abac20" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 45273 } } peers { permanent_uuid: "0c3eb0a5a11d4013ad3576702e194fbd" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39329 } } peers { permanent_uuid: "0ec7637d1cd643f79d025ba05f198dc6" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40245 } }
I20260929 18:42:31.973472 11900 catalog_manager.cc:5495] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c3629afeb143c891d1118564abac20 (127.7.157.67). New cstate: current_term: 1 leader_uuid: "65c3629afeb143c891d1118564abac20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65c3629afeb143c891d1118564abac20" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 45273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c3eb0a5a11d4013ad3576702e194fbd" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ec7637d1cd643f79d025ba05f198dc6" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40245 } health_report { overall_health: UNKNOWN } } }
I20260929 18:42:32.016556 12370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.012987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.7.157.67:45273)
0929 18:42:32.013104 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:32.013109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:32.013123 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:42:32.013274 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:32.013277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:32.013332 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:32.013424 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:32.013676 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:32.014218 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:32.015119 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:32.015329 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:32.015416 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:32.015729 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:32.016152 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:32.016263 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:32.016336 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:42:32.016394 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:42:32.022267 12178 raft_consensus.cc:1261] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd [term 1 FOLLOWER]: Refusing update from remote peer 65c3629afeb143c891d1118564abac20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:32.022267 12045 raft_consensus.cc:1261] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6 [term 1 FOLLOWER]: Refusing update from remote peer 65c3629afeb143c891d1118564abac20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:32.022929 12384 consensus_queue.cc:1035] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c3eb0a5a11d4013ad3576702e194fbd" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:32.023083 12380 consensus_queue.cc:1035] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec7637d1cd643f79d025ba05f198dc6" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:32.030118 12388 mvcc.cc:205] Tried to move back new op lower bound from 7334737313876140032 to 7334737313673805824. Current Snapshot: MvccSnapshot[applied={T|T < 7334737313876140032}]
I20260929 18:42:32.031207 12389 mvcc.cc:205] Tried to move back new op lower bound from 7334737313876140032 to 7334737313673805824. Current Snapshot: MvccSnapshot[applied={T|T < 7334737313876140032}]
I20260929 18:42:32.039983 11899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:32.040184 11899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:32.042722 11899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1136f3e221d4241bc00cd89c78d825c: Sending DeleteTablet for 3 replicas of tablet 5d3c2fc5acfd48c696d189694c2a2351
I20260929 18:42:32.043586 12025 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d3c2fc5acfd48c696d189694c2a2351 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:32 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:55826
I20260929 18:42:32.043586 12158 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d3c2fc5acfd48c696d189694c2a2351 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:32 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:56694
I20260929 18:42:32.043587 12291 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d3c2fc5acfd48c696d189694c2a2351 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:32 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:52466
I20260929 18:42:32.044134 12400 tablet_replica.cc:330] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6: stopping tablet replica
I20260929 18:42:32.044349 12398 tablet_replica.cc:330] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20: stopping tablet replica
I20260929 18:42:32.044492 12400 raft_consensus.cc:2229] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:32.044744 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 11963
I20260929 18:42:32.044762 12398 raft_consensus.cc:2229] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:42:32.045033 12400 raft_consensus.cc:2258] T 5d3c2fc5acfd48c696d189694c2a2351 P 0ec7637d1cd643f79d025ba05f198dc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:32.045290 12398 raft_consensus.cc:2258] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:32.047346 12398 ts_tablet_manager.cc:1905] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:32.049733 12399 tablet_replica.cc:330] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd: stopping tablet replica
I20260929 18:42:32.049789 12398 ts_tablet_manager.cc:1918] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:32.049872 12398 log.cc:1196] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/5d3c2fc5acfd48c696d189694c2a2351
I20260929 18:42:32.050073 12399 raft_consensus.cc:2229] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:32.050182 12398 ts_tablet_manager.cc:1939] T 5d3c2fc5acfd48c696d189694c2a2351 P 65c3629afeb143c891d1118564abac20: Deleting consensus metadata
I20260929 18:42:32.050436 12399 raft_consensus.cc:2258] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:32.051508 11887 catalog_manager.cc:4841] TS 65c3629afeb143c891d1118564abac20 (127.7.157.67:45273): tablet 5d3c2fc5acfd48c696d189694c2a2351 (table test-table [id=b5f4b3bf414948ee82ed4d9473b7f45b]) successfully deleted
I20260929 18:42:32.051874 12399 ts_tablet_manager.cc:1905] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:42:32.053478 11886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:40245 (error 108)
I20260929 18:42:32.054224 12399 ts_tablet_manager.cc:1918] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:32.054306 12399 log.cc:1196] T 5d3c2fc5acfd48c696d189694c2a2351 P 0c3eb0a5a11d4013ad3576702e194fbd: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/5d3c2fc5acfd48c696d189694c2a2351
I20260929 18:42:32.054519 12371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.054214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:40245 (local address 127.0.0.1:55830)
0929 18:42:32.054363 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:32.054433 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:40245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 18:42:32.054545 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 12096
W20260929 18:42:32.055233 11886 catalog_manager.cc:4584] TS 0ec7637d1cd643f79d025ba05f198dc6 (127.7.157.65:40245): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d3c2fc5acfd48c696d189694c2a2351: Network error: Client connection negotiation failed: client connection to 127.7.157.65:40245: connect: Connection refused (error 111)
I20260929 18:42:32.059604 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 12229
I20260929 18:42:32.061288 12371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.059649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:39329 (local address 127.0.0.1:56700)
0929 18:42:32.061160 (+ 1511us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:32.061213 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.66:39329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1430}
I20260929 18:42:32.064327 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 11870
2026-09-29T18:42:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2471 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T18:42:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:32Z Disabled control of system clock
I20260929 18:42:32.091368 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:45085
--webserver_interface=127.7.157.126
--webserver_port=0
--builtin_ntp_servers=127.7.157.84:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:45085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:42:32.196880 12405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:32.197142 12405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:32.197178 12405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:32.201103 12405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:32.201162 12405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:32.201175 12405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:32.201190 12405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:32.201207 12405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:32.205946 12405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:45085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:45085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:32.206946 12405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:32.208331 12405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:42:32.216333 12405 server_base.cc:1034] running on GCE node
W20260929 18:42:32.216370 12410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:32.216409 12411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:32.216498 12413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:32.217082 12405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:32.218160 12405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:32.219347 12405 hybrid_clock.cc:648] HybridClock initialized: now 1790707352219318 us; error 40 us; skew 500 ppm
I20260929 18:42:32.221103 12405 webserver.cc:460] Webserver started at http://127.7.157.126:40301/ using document root <none> and password file <none>
I20260929 18:42:32.221654 12405 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:32.221729 12405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:32.221912 12405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:32.223981 12405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "412ef9ea23834d3aa16d18954ba64110"
format_stamp: "Formatted at 2026-09-29 18:42:32 on dist-test-slave-smpn"
server_key: "6b9d0241012c4a4e2dc9abd33299a3ba"
server_key_iv: "5b8d6ed5af4aa98cf8b9d1515bd9e355"
server_key_version: "encryptionkey@0"
I20260929 18:42:32.224507 12405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "412ef9ea23834d3aa16d18954ba64110"
format_stamp: "Formatted at 2026-09-29 18:42:32 on dist-test-slave-smpn"
server_key: "6b9d0241012c4a4e2dc9abd33299a3ba"
server_key_iv: "5b8d6ed5af4aa98cf8b9d1515bd9e355"
server_key_version: "encryptionkey@0"
I20260929 18:42:32.228103 12405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:42:32.230409 12419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:32.231665 12405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:32.231776 12405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "412ef9ea23834d3aa16d18954ba64110"
format_stamp: "Formatted at 2026-09-29 18:42:32 on dist-test-slave-smpn"
server_key: "6b9d0241012c4a4e2dc9abd33299a3ba"
server_key_iv: "5b8d6ed5af4aa98cf8b9d1515bd9e355"
server_key_version: "encryptionkey@0"
I20260929 18:42:32.231871 12405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:32.236296 12405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:32.236871 12405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:32.237020 12405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:32.243374 12405 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:45085
I20260929 18:42:32.243389 12471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:45085 every 8 connection(s)
I20260929 18:42:32.244266 12405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:32.246150 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 12405
I20260929 18:42:32.246249 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:32.246496 7797 external_mini_cluster.cc:1428] Setting key 41b7286b2b06606407e381f918b38990
I20260929 18:42:32.247364 12472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:32.252262 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.247793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.7.157.126:45085)
0929 18:42:32.248427 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:32.248446 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:32.248487 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:42:32.249305 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:32.249330 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:32.249770 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:32.250121 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:32.250790 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:32.250796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:32.250810 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:42:32.250820 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:32.250881 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:42:32.250904 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:32.251041 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:32.251137 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 18:42:32.251379 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":97,"threads_started":1}
I20260929 18:42:32.254014 12472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110: Bootstrap starting.
I20260929 18:42:32.256150 12472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:32.256819 12472 log.cc:824] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:32.258533 12472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110: No bootstrap required, opened a new log
I20260929 18:42:32.261917 12472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412ef9ea23834d3aa16d18954ba64110" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45085 } }
I20260929 18:42:32.262135 12472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:32.262176 12472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412ef9ea23834d3aa16d18954ba64110, State: Initialized, Role: FOLLOWER
I20260929 18:42:32.262542 12472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412ef9ea23834d3aa16d18954ba64110" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45085 } }
I20260929 18:42:32.262648 12472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:32.262701 12472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:32.262770 12472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:32.263617 12472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412ef9ea23834d3aa16d18954ba64110" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45085 } }
I20260929 18:42:32.263909 12472 leader_election.cc:302] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 412ef9ea23834d3aa16d18954ba64110; no voters:
I20260929 18:42:32.264155 12472 leader_election.cc:288] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:32.264264 12477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:32.264439 12477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [term 1 LEADER]: Becoming Leader. State: Replica: 412ef9ea23834d3aa16d18954ba64110, State: Running, Role: LEADER
I20260929 18:42:32.264721 12477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412ef9ea23834d3aa16d18954ba64110" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45085 } }
I20260929 18:42:32.265429 12472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:32.266328 12478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412ef9ea23834d3aa16d18954ba64110. Latest consensus state: current_term: 1 leader_uuid: "412ef9ea23834d3aa16d18954ba64110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412ef9ea23834d3aa16d18954ba64110" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45085 } } }
I20260929 18:42:32.266449 12478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:32.266702 12479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "412ef9ea23834d3aa16d18954ba64110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412ef9ea23834d3aa16d18954ba64110" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45085 } } }
I20260929 18:42:32.266786 12479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:32.267477 12486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:32.270378 12486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:32.276136 12486 catalog_manager.cc:1347] Generated new cluster ID: ed54438cf739418a8b44153030c6cadd
I20260929 18:42:32.276227 12486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:32.289376 12486 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:32.289677 12486 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:32.312917 12486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110: Generated new TSK 0
I20260929 18:42:32.313848 12486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:42:32.323261 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45085
--builtin_ntp_servers=127.7.157.84:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:42:32.432258 12496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:32.432515 12496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:32.432554 12496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:32.436638 12496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:32.436703 12496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:32.436759 12496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:42:32.441816 12496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:32.442835 12496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:32.444211 12496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:32.453540 12501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:32.453547 12502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:32.453733 12496 server_base.cc:1034] running on GCE node
W20260929 18:42:32.453572 12504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:32.454243 12496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:32.454886 12496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:32.456033 12496 hybrid_clock.cc:648] HybridClock initialized: now 1790707352455997 us; error 48 us; skew 500 ppm
I20260929 18:42:32.457726 12496 webserver.cc:460] Webserver started at http://127.7.157.65:39203/ using document root <none> and password file <none>
I20260929 18:42:32.458319 12496 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:32.458387 12496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:32.458590 12496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:32.460680 12496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "f14e48c604de461990a449929834a0ea"
format_stamp: "Formatted at 2026-09-29 18:42:32 on dist-test-slave-smpn"
server_key: "30b9f3815ffcbad75f7c32785408be24"
server_key_iv: "8da54e742830c5f70dd53e92c8cba578"
server_key_version: "encryptionkey@0"
I20260929 18:42:32.461207 12496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "f14e48c604de461990a449929834a0ea"
format_stamp: "Formatted at 2026-09-29 18:42:32 on dist-test-slave-smpn"
server_key: "30b9f3815ffcbad75f7c32785408be24"
server_key_iv: "8da54e742830c5f70dd53e92c8cba578"
server_key_version: "encryptionkey@0"
I20260929 18:42:32.464823 12496 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:42:32.467216 12510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:32.468443 12496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:32.468541 12496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "f14e48c604de461990a449929834a0ea"
format_stamp: "Formatted at 2026-09-29 18:42:32 on dist-test-slave-smpn"
server_key: "30b9f3815ffcbad75f7c32785408be24"
server_key_iv: "8da54e742830c5f70dd53e92c8cba578"
server_key_version: "encryptionkey@0"
I20260929 18:42:32.468636 12496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:32.473052 12496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:32.473639 12496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:32.473798 12496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:32.474341 12496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:32.475260 12496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:32.475308 12496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:32.475356 12496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:32.475383 12496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:32.483349 12496 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:36353
I20260929 18:42:32.483382 12623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:36353 every 8 connection(s)
I20260929 18:42:32.484282 12496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:42:32.488830 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 12496
I20260929 18:42:32.488925 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:42:32.489157 7797 external_mini_cluster.cc:1428] Setting key 1a93d9ab75d690fd755618527e22940e
I20260929 18:42:32.490207 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.485986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:35119 (local address 127.7.157.126:45085)
0929 18:42:32.486114 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:32.486118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:32.487234 (+ 1116us) server_negotiation.cc:407] Connection header received
0929 18:42:32.488506 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:32.488509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:32.488553 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:32.488651 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:32.489755 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:32.489759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:32.489761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:42:32.489773 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:32.489797 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:42:32.489807 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:32.489889 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:32.490037 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 18:42:32.490074 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:42:32.491161 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.486402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45085 (local address 127.7.157.65:35119)
0929 18:42:32.487055 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:32.487135 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:32.488297 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:32.488841 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:32.488848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:32.488960 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:32.489508 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:42:32.489509 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:32.489552 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:42:32.489623 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:32.489888 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:32.489896 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:32.489901 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:42:32.490076 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 18:42:32.490389 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":99,"threads_started":1}
I20260929 18:42:32.491233 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45085
--builtin_ntp_servers=127.7.157.84:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 18:42:32.492312 12624 heartbeater.cc:347] Connected to a master server at 127.7.157.126:45085
I20260929 18:42:32.492530 12624 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:32.492933 12624 heartbeater.cc:511] Master 127.7.157.126:45085 requested a full tablet report, sending...
I20260929 18:42:32.494366 12436 ts_manager.cc:194] Registered new tserver with Master: f14e48c604de461990a449929834a0ea (127.7.157.65:36353)
W20260929 18:42:32.599243 12627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:32.599516 12627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:32.599556 12627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:32.603646 12627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:32.603710 12627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:32.603786 12627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:42:32.608703 12627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:32.609788 12627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:32.611177 12627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:32.620575 12633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:32.620560 12635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:32.620668 12627 server_base.cc:1034] running on GCE node
W20260929 18:42:32.620556 12632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:32.621160 12627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:32.621829 12627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:32.622982 12627 hybrid_clock.cc:648] HybridClock initialized: now 1790707352622959 us; error 37 us; skew 500 ppm
I20260929 18:42:32.624742 12627 webserver.cc:460] Webserver started at http://127.7.157.66:42207/ using document root <none> and password file <none>
I20260929 18:42:32.625304 12627 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:32.625375 12627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:32.625563 12627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:32.627853 12627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "aaf7f871103741a39aac80e26b7dd79c"
format_stamp: "Formatted at 2026-09-29 18:42:32 on dist-test-slave-smpn"
server_key: "6f9e681db3bf62dd8f8a862da43d9c1d"
server_key_iv: "f92699cbecb4f920f05d2eed1a79bad8"
server_key_version: "encryptionkey@0"
I20260929 18:42:32.628453 12627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "aaf7f871103741a39aac80e26b7dd79c"
format_stamp: "Formatted at 2026-09-29 18:42:32 on dist-test-slave-smpn"
server_key: "6f9e681db3bf62dd8f8a862da43d9c1d"
server_key_iv: "f92699cbecb4f920f05d2eed1a79bad8"
server_key_version: "encryptionkey@0"
I20260929 18:42:32.632310 12627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:42:32.634872 12641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:32.636250 12627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:32.636379 12627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "aaf7f871103741a39aac80e26b7dd79c"
format_stamp: "Formatted at 2026-09-29 18:42:32 on dist-test-slave-smpn"
server_key: "6f9e681db3bf62dd8f8a862da43d9c1d"
server_key_iv: "f92699cbecb4f920f05d2eed1a79bad8"
server_key_version: "encryptionkey@0"
I20260929 18:42:32.636487 12627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:32.641186 12627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:32.641824 12627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:32.641984 12627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:32.642554 12627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:32.643512 12627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:32.643558 12627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:32.643620 12627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:32.643648 12627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:32.652424 12627 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:38551
I20260929 18:42:32.652443 12754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:38551 every 8 connection(s)
I20260929 18:42:32.653450 12627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:42:32.657203 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 12627
I20260929 18:42:32.657323 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:42:32.657634 7797 external_mini_cluster.cc:1428] Setting key 45b44237999548f7a5a0ac078e17b637
I20260929 18:42:32.659647 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45085
--builtin_ntp_servers=127.7.157.84:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 18:42:32.659904 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.655287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:54339 (local address 127.7.157.126:45085)
0929 18:42:32.655425 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:32.655429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:32.656649 (+ 1220us) server_negotiation.cc:407] Connection header received
0929 18:42:32.658183 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:32.658186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:32.658233 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:32.658311 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:32.659466 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:32.659470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:32.659472 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:42:32.659486 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:32.659511 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:42:32.659522 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:32.659612 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:32.659738 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 18:42:32.659784 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:42:32.660820 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.655767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45085 (local address 127.7.157.66:54339)
0929 18:42:32.656493 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:32.656536 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:32.657941 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:32.658497 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:32.658505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:32.658601 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:32.659199 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:42:32.659200 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:32.659244 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:42:32.659317 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:32.659608 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:32.659613 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:32.659617 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:42:32.659769 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 18:42:32.660046 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":112,"threads_started":1}
I20260929 18:42:32.661944 12755 heartbeater.cc:347] Connected to a master server at 127.7.157.126:45085
I20260929 18:42:32.662182 12755 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:32.662640 12755 heartbeater.cc:511] Master 127.7.157.126:45085 requested a full tablet report, sending...
I20260929 18:42:32.663792 12436 ts_manager.cc:194] Registered new tserver with Master: aaf7f871103741a39aac80e26b7dd79c (127.7.157.66:38551)
W20260929 18:42:32.768985 12758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:32.769250 12758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:32.769289 12758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:32.773365 12758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:32.773417 12758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:32.773474 12758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:42:32.778173 12758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:32.779186 12758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:32.780465 12758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:32.789510 12763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:32.789597 12766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:32.789739 12758 server_base.cc:1034] running on GCE node
W20260929 18:42:32.789707 12764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:32.790309 12758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:32.790947 12758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:32.792097 12758 hybrid_clock.cc:648] HybridClock initialized: now 1790707352792070 us; error 40 us; skew 500 ppm
I20260929 18:42:32.794064 12758 webserver.cc:460] Webserver started at http://127.7.157.67:39257/ using document root <none> and password file <none>
I20260929 18:42:32.794611 12758 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:32.794672 12758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:32.794883 12758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:32.796890 12758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "1a3e9f4051cc49069002e972e9163648"
format_stamp: "Formatted at 2026-09-29 18:42:32 on dist-test-slave-smpn"
server_key: "ad3490104e2d42d51065a0c302649ccb"
server_key_iv: "a13dc286443b83bc4692bd5355aa790f"
server_key_version: "encryptionkey@0"
I20260929 18:42:32.797384 12758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "1a3e9f4051cc49069002e972e9163648"
format_stamp: "Formatted at 2026-09-29 18:42:32 on dist-test-slave-smpn"
server_key: "ad3490104e2d42d51065a0c302649ccb"
server_key_iv: "a13dc286443b83bc4692bd5355aa790f"
server_key_version: "encryptionkey@0"
I20260929 18:42:32.800953 12758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:42:32.803231 12772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:32.804378 12758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:32.804486 12758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "1a3e9f4051cc49069002e972e9163648"
format_stamp: "Formatted at 2026-09-29 18:42:32 on dist-test-slave-smpn"
server_key: "ad3490104e2d42d51065a0c302649ccb"
server_key_iv: "a13dc286443b83bc4692bd5355aa790f"
server_key_version: "encryptionkey@0"
I20260929 18:42:32.804572 12758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:32.808940 12758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:32.809505 12758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:32.809655 12758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:32.810149 12758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:32.811009 12758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:32.811054 12758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:32.811153 12758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:32.811192 12758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:32.819439 12758 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:40121
I20260929 18:42:32.819461 12885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:40121 every 8 connection(s)
I20260929 18:42:32.820324 12758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:42:32.825413 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 12758
I20260929 18:42:32.825521 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:42:32.825763 7797 external_mini_cluster.cc:1428] Setting key 871eba3a640768ff3a4f8ae9284eb6e1
I20260929 18:42:32.826107 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.821988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:53699 (local address 127.7.157.126:45085)
0929 18:42:32.822103 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:32.822106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:32.823203 (+ 1097us) server_negotiation.cc:407] Connection header received
0929 18:42:32.824484 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:32.824487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:32.824533 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:32.824610 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:32.825661 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:32.825664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:32.825667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:42:32.825688 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:32.825711 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:42:32.825719 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:32.825798 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:32.825931 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 18:42:32.825974 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:42:32.827118 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.822408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45085 (local address 127.7.157.67:53699)
0929 18:42:32.823033 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:32.823109 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:32.824277 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:32.824779 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:32.824787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:32.824878 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:32.825404 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:42:32.825406 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:32.825446 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:42:32.825517 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:32.825789 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:32.825796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:32.825802 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:42:32.825966 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 18:42:32.826294 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
I20260929 18:42:32.828063 12886 heartbeater.cc:347] Connected to a master server at 127.7.157.126:45085
I20260929 18:42:32.828265 12886 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:32.828622 12886 heartbeater.cc:511] Master 127.7.157.126:45085 requested a full tablet report, sending...
I20260929 18:42:32.829536 12436 ts_manager.cc:194] Registered new tserver with Master: 1a3e9f4051cc49069002e972e9163648 (127.7.157.67:40121)
I20260929 18:42:32.839473 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:42:32.842792 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.841294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.7.157.126:45085)
0929 18:42:32.841432 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:32.841437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:32.841601 (+ 164us) server_negotiation.cc:407] Connection header received
0929 18:42:32.841731 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:32.841734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:32.841771 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:32.841836 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:32.842441 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:32.842445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:32.842448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:42:32.842459 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:32.842480 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:42:32.842487 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:32.842557 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:32.842627 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 18:42:32.842666 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:42:32.845417 12436 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:42:32.848461 12436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:32.860637 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.857402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:36353 (local address 127.0.0.1:59180)
0929 18:42:32.857837 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:32.857887 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:32.857995 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:32.859087 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:32.859094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:32.859171 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:32.859970 (+ 799us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:42:32.859972 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:32.859982 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:42:32.860003 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:32.860288 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:32.860297 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:32.860303 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:42:32.860420 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 18:42:32.860489 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":126,"threads_started":1}
I20260929 18:42:32.860716 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.858181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59180 (local address 127.7.157.65:36353)
0929 18:42:32.858580 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:32.858593 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:32.858617 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:42:32.858685 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:32.858692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:32.858909 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:32.859050 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:32.860112 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:32.860117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:32.860127 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:42:32.860138 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:32.860177 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:42:32.860204 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:32.860293 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:32.860462 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 18:42:32.860539 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":97,"threads_started":1}
I20260929 18:42:32.860783 12898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.857529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:38551 (local address 127.0.0.1:55216)
0929 18:42:32.857919 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:32.857932 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:32.858003 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:32.859328 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:32.859331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:32.859344 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:32.859782 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:42:32.859784 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:32.859803 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:42:32.859868 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:32.860604 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:32.860606 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:32.860610 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:42:32.860658 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 18:42:32.860688 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":113,"threads_started":1}
I20260929 18:42:32.861210 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.858144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.7.157.66:38551)
0929 18:42:32.858875 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:32.858886 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:32.858910 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:42:32.858972 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:32.858978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:32.859187 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:32.859530 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:32.860409 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:32.860415 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:32.860426 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:42:32.860435 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:32.860484 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:42:32.860510 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:32.860945 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:32.860990 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 18:42:32.861056 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":131,"threads_started":1}
I20260929 18:42:32.861706 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.858174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:40121 (local address 127.0.0.1:60602)
0929 18:42:32.858560 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:32.858573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:32.858644 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:32.859772 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:32.859775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:32.859790 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:32.860341 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:42:32.860342 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:32.860350 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:42:32.860369 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:32.861475 (+ 1106us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:32.861478 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:32.861482 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:42:32.861549 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:42:32.861582 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":107,"threads_started":1}
I20260929 18:42:32.862179 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.858307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60602 (local address 127.7.157.67:40121)
0929 18:42:32.858869 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:32.858883 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:32.858905 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:42:32.858954 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:32.858959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:32.859154 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:32.859282 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:32.860493 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:32.860499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:32.860510 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:42:32.860521 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:32.860561 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:42:32.860593 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:32.861910 (+ 1317us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:32.861966 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 18:42:32.862027 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":85,"threads_started":1}
I20260929 18:42:32.863687 12558 tablet_service.cc:1467] Processing CreateTablet for tablet a29c4ace958f4db4a8fc8ba1258b1bc6 (DEFAULT_TABLE table=test-table [id=f32f42a3980c4bd68eea7c66f8be7549]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:32.863749 12689 tablet_service.cc:1467] Processing CreateTablet for tablet a29c4ace958f4db4a8fc8ba1258b1bc6 (DEFAULT_TABLE table=test-table [id=f32f42a3980c4bd68eea7c66f8be7549]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:32.864650 12558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a29c4ace958f4db4a8fc8ba1258b1bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:32.864732 12689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a29c4ace958f4db4a8fc8ba1258b1bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:32.864732 12820 tablet_service.cc:1467] Processing CreateTablet for tablet a29c4ace958f4db4a8fc8ba1258b1bc6 (DEFAULT_TABLE table=test-table [id=f32f42a3980c4bd68eea7c66f8be7549]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:32.865664 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a29c4ace958f4db4a8fc8ba1258b1bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:32.871182 12903 tablet_bootstrap.cc:492] T a29c4ace958f4db4a8fc8ba1258b1bc6 P f14e48c604de461990a449929834a0ea: Bootstrap starting.
I20260929 18:42:32.872236 12904 tablet_bootstrap.cc:492] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c: Bootstrap starting.
I20260929 18:42:32.873147 12905 tablet_bootstrap.cc:492] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648: Bootstrap starting.
I20260929 18:42:32.873353 12903 tablet_bootstrap.cc:654] T a29c4ace958f4db4a8fc8ba1258b1bc6 P f14e48c604de461990a449929834a0ea: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:32.874183 12904 tablet_bootstrap.cc:654] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:32.874217 12903 log.cc:824] T a29c4ace958f4db4a8fc8ba1258b1bc6 P f14e48c604de461990a449929834a0ea: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:32.874919 12904 log.cc:824] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:32.875249 12905 tablet_bootstrap.cc:654] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:32.875999 12905 log.cc:824] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:32.876279 12903 tablet_bootstrap.cc:492] T a29c4ace958f4db4a8fc8ba1258b1bc6 P f14e48c604de461990a449929834a0ea: No bootstrap required, opened a new log
I20260929 18:42:32.876458 12903 ts_tablet_manager.cc:1397] T a29c4ace958f4db4a8fc8ba1258b1bc6 P f14e48c604de461990a449929834a0ea: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:42:32.876775 12904 tablet_bootstrap.cc:492] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c: No bootstrap required, opened a new log
I20260929 18:42:32.876964 12904 ts_tablet_manager.cc:1397] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:42:32.877923 12905 tablet_bootstrap.cc:492] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648: No bootstrap required, opened a new log
I20260929 18:42:32.878091 12905 ts_tablet_manager.cc:1397] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:42:32.879899 12903 raft_consensus.cc:348] T a29c4ace958f4db4a8fc8ba1258b1bc6 P f14e48c604de461990a449929834a0ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14e48c604de461990a449929834a0ea" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36353 } } peers { permanent_uuid: "aaf7f871103741a39aac80e26b7dd79c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38551 } } peers { permanent_uuid: "1a3e9f4051cc49069002e972e9163648" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40121 } }
I20260929 18:42:32.880120 12903 raft_consensus.cc:374] T a29c4ace958f4db4a8fc8ba1258b1bc6 P f14e48c604de461990a449929834a0ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:32.880172 12903 raft_consensus.cc:729] T a29c4ace958f4db4a8fc8ba1258b1bc6 P f14e48c604de461990a449929834a0ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f14e48c604de461990a449929834a0ea, State: Initialized, Role: FOLLOWER
I20260929 18:42:32.880627 12903 consensus_queue.cc:260] T a29c4ace958f4db4a8fc8ba1258b1bc6 P f14e48c604de461990a449929834a0ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14e48c604de461990a449929834a0ea" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36353 } } peers { permanent_uuid: "aaf7f871103741a39aac80e26b7dd79c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38551 } } peers { permanent_uuid: "1a3e9f4051cc49069002e972e9163648" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40121 } }
I20260929 18:42:32.881210 12904 raft_consensus.cc:348] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14e48c604de461990a449929834a0ea" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36353 } } peers { permanent_uuid: "aaf7f871103741a39aac80e26b7dd79c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38551 } } peers { permanent_uuid: "1a3e9f4051cc49069002e972e9163648" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40121 } }
I20260929 18:42:32.881459 12904 raft_consensus.cc:374] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:32.881520 12904 raft_consensus.cc:729] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf7f871103741a39aac80e26b7dd79c, State: Initialized, Role: FOLLOWER
I20260929 18:42:32.881649 12903 ts_tablet_manager.cc:1428] T a29c4ace958f4db4a8fc8ba1258b1bc6 P f14e48c604de461990a449929834a0ea: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260929 18:42:32.881778 12624 heartbeater.cc:503] Master 127.7.157.126:45085 was elected leader, sending a full tablet report...
I20260929 18:42:32.881942 12904 consensus_queue.cc:260] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14e48c604de461990a449929834a0ea" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36353 } } peers { permanent_uuid: "aaf7f871103741a39aac80e26b7dd79c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38551 } } peers { permanent_uuid: "1a3e9f4051cc49069002e972e9163648" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40121 } }
I20260929 18:42:32.882269 12905 raft_consensus.cc:348] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14e48c604de461990a449929834a0ea" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36353 } } peers { permanent_uuid: "aaf7f871103741a39aac80e26b7dd79c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38551 } } peers { permanent_uuid: "1a3e9f4051cc49069002e972e9163648" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40121 } }
I20260929 18:42:32.882493 12905 raft_consensus.cc:374] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:32.882534 12905 raft_consensus.cc:729] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a3e9f4051cc49069002e972e9163648, State: Initialized, Role: FOLLOWER
I20260929 18:42:32.882592 12755 heartbeater.cc:503] Master 127.7.157.126:45085 was elected leader, sending a full tablet report...
I20260929 18:42:32.882915 12904 ts_tablet_manager.cc:1428] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:42:32.882949 12905 consensus_queue.cc:260] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14e48c604de461990a449929834a0ea" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36353 } } peers { permanent_uuid: "aaf7f871103741a39aac80e26b7dd79c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38551 } } peers { permanent_uuid: "1a3e9f4051cc49069002e972e9163648" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40121 } }
I20260929 18:42:32.883522 12886 heartbeater.cc:503] Master 127.7.157.126:45085 was elected leader, sending a full tablet report...
I20260929 18:42:32.883863 12905 ts_tablet_manager.cc:1428] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260929 18:42:32.892165 12911 raft_consensus.cc:482] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:32.892318 12911 raft_consensus.cc:504] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14e48c604de461990a449929834a0ea" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36353 } } peers { permanent_uuid: "aaf7f871103741a39aac80e26b7dd79c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38551 } } peers { permanent_uuid: "1a3e9f4051cc49069002e972e9163648" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40121 } }
I20260929 18:42:32.893285 12911 leader_election.cc:288] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f14e48c604de461990a449929834a0ea (127.7.157.65:36353), aaf7f871103741a39aac80e26b7dd79c (127.7.157.66:38551)
I20260929 18:42:32.894899 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.893583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:38263 (local address 127.7.157.66:38551)
0929 18:42:32.893703 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:32.893707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:32.893724 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:42:32.893763 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:32.893766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:32.893798 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:32.893861 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:32.894569 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:32.894572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:32.894574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:42:32.894584 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:32.894601 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:42:32.894609 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:32.894695 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:32.894748 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 18:42:32.894783 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:42:32.894899 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.893513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:38551 (local address 127.7.157.67:38263)
0929 18:42:32.893613 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:32.893628 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:32.893693 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:32.893907 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:32.893910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:32.893924 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:32.894425 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:42:32.894427 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:32.894441 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:42:32.894465 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:32.894675 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:32.894678 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:32.894681 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:42:32.894750 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:42:32.894783 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 18:42:32.895272 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.893574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:55285 (local address 127.7.157.65:36353)
0929 18:42:32.893721 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:32.893725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:32.894149 (+ 424us) server_negotiation.cc:407] Connection header received
0929 18:42:32.894217 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:32.894219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:32.894250 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:32.894325 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:32.894849 (+ 524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:32.894852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:32.894855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:42:32.894866 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:32.894885 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:42:32.894893 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:32.894963 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:32.895123 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 18:42:32.895159 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 18:42:32.895289 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.893522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:36353 (local address 127.7.157.67:55285)
0929 18:42:32.894044 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:32.894055 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:32.894143 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:32.894362 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:32.894365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:32.894379 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:32.894717 (+ 338us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:42:32.894719 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:32.894728 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:42:32.894749 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:32.894956 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:32.894959 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:32.894962 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:42:32.895126 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 18:42:32.895157 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":216,"thread_start_us":124,"threads_started":1}
I20260929 18:42:32.895638 12709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a29c4ace958f4db4a8fc8ba1258b1bc6" candidate_uuid: "1a3e9f4051cc49069002e972e9163648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf7f871103741a39aac80e26b7dd79c" is_pre_election: true
I20260929 18:42:32.895912 12709 raft_consensus.cc:2454] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a3e9f4051cc49069002e972e9163648 in term 0.
I20260929 18:42:32.896381 12774 leader_election.cc:302] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a3e9f4051cc49069002e972e9163648, aaf7f871103741a39aac80e26b7dd79c; no voters:
I20260929 18:42:32.896562 12578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a29c4ace958f4db4a8fc8ba1258b1bc6" candidate_uuid: "1a3e9f4051cc49069002e972e9163648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f14e48c604de461990a449929834a0ea" is_pre_election: true
I20260929 18:42:32.896657 12911 raft_consensus.cc:2781] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:32.896751 12911 raft_consensus.cc:482] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:32.896799 12911 raft_consensus.cc:3037] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:32.896835 12578 raft_consensus.cc:2454] T a29c4ace958f4db4a8fc8ba1258b1bc6 P f14e48c604de461990a449929834a0ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a3e9f4051cc49069002e972e9163648 in term 0.
I20260929 18:42:32.897796 12911 raft_consensus.cc:504] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14e48c604de461990a449929834a0ea" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36353 } } peers { permanent_uuid: "aaf7f871103741a39aac80e26b7dd79c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38551 } } peers { permanent_uuid: "1a3e9f4051cc49069002e972e9163648" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40121 } }
I20260929 18:42:32.898104 12911 leader_election.cc:288] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [CANDIDATE]: Term 1 election: Requested vote from peers f14e48c604de461990a449929834a0ea (127.7.157.65:36353), aaf7f871103741a39aac80e26b7dd79c (127.7.157.66:38551)
I20260929 18:42:32.898379 12578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a29c4ace958f4db4a8fc8ba1258b1bc6" candidate_uuid: "1a3e9f4051cc49069002e972e9163648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f14e48c604de461990a449929834a0ea"
I20260929 18:42:32.898411 12709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a29c4ace958f4db4a8fc8ba1258b1bc6" candidate_uuid: "1a3e9f4051cc49069002e972e9163648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf7f871103741a39aac80e26b7dd79c"
I20260929 18:42:32.898488 12578 raft_consensus.cc:3037] T a29c4ace958f4db4a8fc8ba1258b1bc6 P f14e48c604de461990a449929834a0ea [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:32.898533 12709 raft_consensus.cc:3037] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:32.899597 12578 raft_consensus.cc:2454] T a29c4ace958f4db4a8fc8ba1258b1bc6 P f14e48c604de461990a449929834a0ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a3e9f4051cc49069002e972e9163648 in term 1.
I20260929 18:42:32.899605 12709 raft_consensus.cc:2454] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a3e9f4051cc49069002e972e9163648 in term 1.
I20260929 18:42:32.899901 12773 leader_election.cc:302] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a3e9f4051cc49069002e972e9163648, f14e48c604de461990a449929834a0ea; no voters:
I20260929 18:42:32.900070 12911 raft_consensus.cc:2781] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:32.900282 12911 raft_consensus.cc:686] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [term 1 LEADER]: Becoming Leader. State: Replica: 1a3e9f4051cc49069002e972e9163648, State: Running, Role: LEADER
I20260929 18:42:32.900604 12911 consensus_queue.cc:237] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14e48c604de461990a449929834a0ea" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36353 } } peers { permanent_uuid: "aaf7f871103741a39aac80e26b7dd79c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38551 } } peers { permanent_uuid: "1a3e9f4051cc49069002e972e9163648" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40121 } }
I20260929 18:42:32.903789 12435 catalog_manager.cc:5495] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a3e9f4051cc49069002e972e9163648 (127.7.157.67). New cstate: current_term: 1 leader_uuid: "1a3e9f4051cc49069002e972e9163648" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14e48c604de461990a449929834a0ea" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf7f871103741a39aac80e26b7dd79c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a3e9f4051cc49069002e972e9163648" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40121 } health_report { overall_health: HEALTHY } } }
W20260929 18:42:32.904342 12756 tablet.cc:2369] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:42:32.911793 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.910559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.7.157.67:40121)
0929 18:42:32.910671 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:32.910675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:32.910691 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:42:32.910816 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:32.910818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:32.910869 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:32.910931 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:32.911491 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:32.911494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:32.911496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:42:32.911506 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:32.911523 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:42:32.911530 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:32.911596 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:32.911639 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 18:42:32.911669 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:42:32.917145 12709 raft_consensus.cc:1261] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c [term 1 FOLLOWER]: Refusing update from remote peer 1a3e9f4051cc49069002e972e9163648: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:32.917146 12578 raft_consensus.cc:1261] T a29c4ace958f4db4a8fc8ba1258b1bc6 P f14e48c604de461990a449929834a0ea [term 1 FOLLOWER]: Refusing update from remote peer 1a3e9f4051cc49069002e972e9163648: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:32.917800 12911 consensus_queue.cc:1035] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f14e48c604de461990a449929834a0ea" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:32.917977 12913 consensus_queue.cc:1035] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf7f871103741a39aac80e26b7dd79c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:32.924546 12916 mvcc.cc:205] Tried to move back new op lower bound from 7334737317542170624 to 7334737317483528192. Current Snapshot: MvccSnapshot[applied={T|T < 7334737317542170624}]
I20260929 18:42:32.924734 12917 mvcc.cc:205] Tried to move back new op lower bound from 7334737317542170624 to 7334737317483528192. Current Snapshot: MvccSnapshot[applied={T|T < 7334737317542170624}]
I20260929 18:42:32.933432 12436 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:32.933581 12436 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:32.935719 12436 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 412ef9ea23834d3aa16d18954ba64110: Sending DeleteTablet for 3 replicas of tablet a29c4ace958f4db4a8fc8ba1258b1bc6
I20260929 18:42:32.936403 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet a29c4ace958f4db4a8fc8ba1258b1bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:32 UTC) from {username='slave'} at 127.0.0.1:60602
I20260929 18:42:32.936479 12558 tablet_service.cc:1514] Processing DeleteTablet for tablet a29c4ace958f4db4a8fc8ba1258b1bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:32 UTC) from {username='slave'} at 127.0.0.1:59180
I20260929 18:42:32.936875 12689 tablet_service.cc:1514] Processing DeleteTablet for tablet a29c4ace958f4db4a8fc8ba1258b1bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:32 UTC) from {username='slave'} at 127.0.0.1:55216
I20260929 18:42:32.936969 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 12496
I20260929 18:42:32.936860 12926 tablet_replica.cc:330] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648: stopping tablet replica
I20260929 18:42:32.938445 12926 raft_consensus.cc:2229] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:42:32.938876 12926 raft_consensus.cc:2258] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:32.940608 12926 ts_tablet_manager.cc:1905] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:32.943264 12928 tablet_replica.cc:330] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c: stopping tablet replica
I20260929 18:42:32.943504 12926 ts_tablet_manager.cc:1918] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:32.943529 12928 raft_consensus.cc:2229] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:32.943653 12926 log.cc:1196] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/a29c4ace958f4db4a8fc8ba1258b1bc6
I20260929 18:42:32.943794 12928 raft_consensus.cc:2258] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:32.944105 12926 ts_tablet_manager.cc:1939] T a29c4ace958f4db4a8fc8ba1258b1bc6 P 1a3e9f4051cc49069002e972e9163648: Deleting consensus metadata
I20260929 18:42:32.945019 12928 ts_tablet_manager.cc:1905] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:32.945341 12420 catalog_manager.cc:4841] TS 1a3e9f4051cc49069002e972e9163648 (127.7.157.67:40121): tablet a29c4ace958f4db4a8fc8ba1258b1bc6 (table test-table [id=f32f42a3980c4bd68eea7c66f8be7549]) successfully deleted
I20260929 18:42:32.947537 12928 ts_tablet_manager.cc:1918] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:32.947633 12928 log.cc:1196] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/a29c4ace958f4db4a8fc8ba1258b1bc6
I20260929 18:42:32.947916 12928 ts_tablet_manager.cc:1939] T a29c4ace958f4db4a8fc8ba1258b1bc6 P aaf7f871103741a39aac80e26b7dd79c: Deleting consensus metadata
I20260929 18:42:32.949127 12421 catalog_manager.cc:4841] TS aaf7f871103741a39aac80e26b7dd79c (127.7.157.66:38551): tablet a29c4ace958f4db4a8fc8ba1258b1bc6 (table test-table [id=f32f42a3980c4bd68eea7c66f8be7549]) successfully deleted
W20260929 18:42:32.949733 12420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:36353 (error 108)
I20260929 18:42:32.949910 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 12627
I20260929 18:42:32.950989 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:32.950679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:36353 (local address 127.0.0.1:59192)
0929 18:42:32.950845 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:32.950911 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260929 18:42:32.951146 12420 catalog_manager.cc:4584] TS f14e48c604de461990a449929834a0ea (127.7.157.65:36353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a29c4ace958f4db4a8fc8ba1258b1bc6: Network error: Client connection negotiation failed: client connection to 127.7.157.65:36353: connect: Connection refused (error 111)
I20260929 18:42:32.957363 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 12758
I20260929 18:42:32.962081 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 12405
2026-09-29T18:42:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (897 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 18:42:32.982033 7797 mini_oidc.cc:152] Starting JWKS server
I20260929 18:42:32.983762 7797 webserver.cc:460] Webserver started at https://127.0.0.1:35317/ using document root <none> and password file <none>
I20260929 18:42:32.983953 7797 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:42:32.984252 7797 webserver.cc:460] Webserver started at http://127.0.0.1:44943/ using document root <none> and password file <none>
2026-09-29T18:42:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:32Z Disabled control of system clock
I20260929 18:42:32.993690 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:44815
--webserver_interface=127.7.157.126
--webserver_port=0
--builtin_ntp_servers=127.7.157.84:36251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:44815
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35317/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/testchainca.cert with env {}
W20260929 18:42:33.103850 12935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:33.104122 12935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:33.104163 12935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:33.108135 12935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:33.108199 12935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:33.108213 12935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:33.108228 12935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:33.108248 12935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:42:33.108263 12935 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:42:33.108280 12935 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35317/jwks.json
I20260929 18:42:33.113718 12935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:36251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:44815
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:44815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35317/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:33.114779 12935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:33.115866 12935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:33.124402 12940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:33.124439 12943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:33.124462 12935 server_base.cc:1034] running on GCE node
W20260929 18:42:33.124418 12941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:33.125120 12935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:33.126224 12935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:33.127374 12935 hybrid_clock.cc:648] HybridClock initialized: now 1790707353127354 us; error 30 us; skew 500 ppm
I20260929 18:42:33.129184 12935 webserver.cc:460] Webserver started at http://127.7.157.126:38345/ using document root <none> and password file <none>
I20260929 18:42:33.129736 12935 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:33.129797 12935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:33.129982 12935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:33.132025 12935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "a62cc56d6bba4272aaea0ec3781f1ef2"
format_stamp: "Formatted at 2026-09-29 18:42:33 on dist-test-slave-smpn"
server_key: "b4c87200577e237109182e849e707d48"
server_key_iv: "c391e80b350c8babd63efebfaf886126"
server_key_version: "encryptionkey@0"
I20260929 18:42:33.132541 12935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "a62cc56d6bba4272aaea0ec3781f1ef2"
format_stamp: "Formatted at 2026-09-29 18:42:33 on dist-test-slave-smpn"
server_key: "b4c87200577e237109182e849e707d48"
server_key_iv: "c391e80b350c8babd63efebfaf886126"
server_key_version: "encryptionkey@0"
I20260929 18:42:33.136101 12935 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:42:33.138379 12949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:33.139621 12935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:33.139740 12935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "a62cc56d6bba4272aaea0ec3781f1ef2"
format_stamp: "Formatted at 2026-09-29 18:42:33 on dist-test-slave-smpn"
server_key: "b4c87200577e237109182e849e707d48"
server_key_iv: "c391e80b350c8babd63efebfaf886126"
server_key_version: "encryptionkey@0"
I20260929 18:42:33.139832 12935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:33.160661 12935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:33.161372 12935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:33.161550 12935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:33.168208 12935 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:44815
I20260929 18:42:33.168232 13001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:44815 every 8 connection(s)
I20260929 18:42:33.169147 12935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:33.169559 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 12935
I20260929 18:42:33.169679 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:33.169989 7797 external_mini_cluster.cc:1428] Setting key 9ee2582a7d54095b233204aeb45a5762
I20260929 18:42:33.172320 13002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:33.178452 13002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2: Bootstrap starting.
I20260929 18:42:33.178915 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:33.171312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.7.157.126:44815)
0929 18:42:33.171898 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:33.171913 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:33.171951 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:42:33.172606 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:33.172624 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:33.173033 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:33.173318 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:33.173651 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.174601 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.175833 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.176293 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.176405 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:33.177226 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:33.177230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:33.177242 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:42:33.177611 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:33.177684 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:42:33.177703 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:33.177774 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:33.177925 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 18:42:33.178181 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":96,"threads_started":1}
I20260929 18:42:33.180846 13002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:33.181615 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:42:33.181639 13002 log.cc:824] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:33.183604 13002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2: No bootstrap required, opened a new log
I20260929 18:42:33.186499 13002 raft_consensus.cc:348] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62cc56d6bba4272aaea0ec3781f1ef2" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44815 } }
I20260929 18:42:33.186719 13002 raft_consensus.cc:374] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:33.186775 13002 raft_consensus.cc:729] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62cc56d6bba4272aaea0ec3781f1ef2, State: Initialized, Role: FOLLOWER
I20260929 18:42:33.187193 13002 consensus_queue.cc:260] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62cc56d6bba4272aaea0ec3781f1ef2" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44815 } }
I20260929 18:42:33.187319 13002 raft_consensus.cc:388] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:33.187381 13002 raft_consensus.cc:482] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:33.187476 13002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:33.188305 13002 raft_consensus.cc:504] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62cc56d6bba4272aaea0ec3781f1ef2" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44815 } }
I20260929 18:42:33.188614 13002 leader_election.cc:302] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a62cc56d6bba4272aaea0ec3781f1ef2; no voters:
I20260929 18:42:33.188884 13002 leader_election.cc:288] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:33.188970 13007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:33.189182 13007 raft_consensus.cc:686] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [term 1 LEADER]: Becoming Leader. State: Replica: a62cc56d6bba4272aaea0ec3781f1ef2, State: Running, Role: LEADER
I20260929 18:42:33.189451 13007 consensus_queue.cc:237] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62cc56d6bba4272aaea0ec3781f1ef2" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44815 } }
I20260929 18:42:33.190023 13002 sys_catalog.cc:564] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:33.190968 13009 sys_catalog.cc:455] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a62cc56d6bba4272aaea0ec3781f1ef2. Latest consensus state: current_term: 1 leader_uuid: "a62cc56d6bba4272aaea0ec3781f1ef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62cc56d6bba4272aaea0ec3781f1ef2" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44815 } } }
I20260929 18:42:33.190963 13008 sys_catalog.cc:455] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a62cc56d6bba4272aaea0ec3781f1ef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62cc56d6bba4272aaea0ec3781f1ef2" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44815 } } }
I20260929 18:42:33.191114 13009 sys_catalog.cc:458] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:33.191114 13008 sys_catalog.cc:458] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:33.191448 13016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:33.194267 13016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:33.199982 13016 catalog_manager.cc:1347] Generated new cluster ID: 6cda883cd2464931b9e808410f7db9f8
I20260929 18:42:33.200045 13016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:33.209609 13016 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:33.210438 13016 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:33.214736 13016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a62cc56d6bba4272aaea0ec3781f1ef2: Generated new TSK 0
I20260929 18:42:33.215399 13016 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 18:42:33.241390 13032 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 18:42:33.241511 13032 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 18:42:33.242003 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:33.225852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.7.157.126:44815)
0929 18:42:33.225975 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:33.225980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:33.226179 (+ 199us) server_negotiation.cc:407] Connection header received
0929 18:42:33.226389 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:33.226391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:33.226436 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:33.226514 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:42:33.226760 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.227297 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.228464 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.228658 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.228734 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:33.229067 (+ 333us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:42:33.241656 (+ 12589us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 18:42:33.241894 (+ 238us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47176: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 18:42:33.242059 13005 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47176: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 18:42:33.243577 13031 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:33.225755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:44815 (local address 127.0.0.1:47176)
0929 18:42:33.226091 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:33.226106 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:33.226247 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:33.226542 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:33.226546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:33.226553 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:42:33.226663 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:33.226669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.227387 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.227391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:33.228362 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:33.228372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.228759 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.228762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:33.228849 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:33.228900 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:42:33.241896 (+ 12996us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 18:42:33.242149 (+ 253us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.157.126:44815: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":89,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.7.157.126:44815: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 18:42:33.244196 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 12935
2026-09-29T18:42:33Z chronyd exiting
I20260929 18:42:33.280735 7797 test_util.cc:175] -----------------------------------------------
I20260929 18:42:33.280819 7797 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707328618849-7797-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (300 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 18:42:33.281777 7797 mini_oidc.cc:152] Starting JWKS server
I20260929 18:42:33.282630 7797 webserver.cc:460] Webserver started at https://127.0.0.1:45023/ using document root <none> and password file <none>
I20260929 18:42:33.282720 7797 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:42:33.282956 7797 webserver.cc:460] Webserver started at http://127.0.0.1:32919/ using document root <none> and password file <none>
2026-09-29T18:42:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:33Z Disabled control of system clock
I20260929 18:42:33.290732 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:39513
--webserver_interface=127.7.157.126
--webserver_port=0
--builtin_ntp_servers=127.7.157.84:36107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:39513
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45023/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/testchainca.cert with env {}
W20260929 18:42:33.398782 13038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:33.399055 13038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:33.399135 13038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:33.403108 13038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:33.403175 13038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:33.403189 13038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:33.403205 13038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:33.403225 13038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:42:33.403240 13038 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:42:33.403258 13038 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45023/jwks.json
I20260929 18:42:33.408598 13038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:36107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:39513
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:39513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45023/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:33.409608 13038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:33.410578 13038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:33.418753 13044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:33.418812 13038 server_base.cc:1034] running on GCE node
W20260929 18:42:33.418762 13043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:33.418776 13046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:33.419543 13038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:33.420612 13038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:33.421777 13038 hybrid_clock.cc:648] HybridClock initialized: now 1790707353421759 us; error 30 us; skew 500 ppm
I20260929 18:42:33.423556 13038 webserver.cc:460] Webserver started at http://127.7.157.126:34359/ using document root <none> and password file <none>
I20260929 18:42:33.424106 13038 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:33.424173 13038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:33.424352 13038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:33.426338 13038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "881cad64f1a14973af76404b585dc6d7"
format_stamp: "Formatted at 2026-09-29 18:42:33 on dist-test-slave-smpn"
server_key: "37429ad7465d40ad7769f420ccce44ee"
server_key_iv: "b46d2d6ce0798e3491b36104e8ff7be2"
server_key_version: "encryptionkey@0"
I20260929 18:42:33.426878 13038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "881cad64f1a14973af76404b585dc6d7"
format_stamp: "Formatted at 2026-09-29 18:42:33 on dist-test-slave-smpn"
server_key: "37429ad7465d40ad7769f420ccce44ee"
server_key_iv: "b46d2d6ce0798e3491b36104e8ff7be2"
server_key_version: "encryptionkey@0"
I20260929 18:42:33.430418 13038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:42:33.432780 13052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:33.434003 13038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:33.434111 13038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "881cad64f1a14973af76404b585dc6d7"
format_stamp: "Formatted at 2026-09-29 18:42:33 on dist-test-slave-smpn"
server_key: "37429ad7465d40ad7769f420ccce44ee"
server_key_iv: "b46d2d6ce0798e3491b36104e8ff7be2"
server_key_version: "encryptionkey@0"
I20260929 18:42:33.434207 13038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:33.448314 13038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:33.449009 13038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:33.449184 13038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:33.456012 13038 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:39513
I20260929 18:42:33.456037 13104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:39513 every 8 connection(s)
I20260929 18:42:33.456939 13038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:33.459998 13105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:33.465508 13105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7: Bootstrap starting.
I20260929 18:42:33.465571 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 13038
I20260929 18:42:33.465667 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:33.465941 7797 external_mini_cluster.cc:1428] Setting key 1d68b0fd6c776a875d43de0ae6e46ec4
I20260929 18:42:33.468258 13105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:33.468966 13105 log.cc:824] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:33.470875 13105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7: No bootstrap required, opened a new log
I20260929 18:42:33.474925 13105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cad64f1a14973af76404b585dc6d7" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39513 } }
I20260929 18:42:33.475211 13105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:33.475268 13105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881cad64f1a14973af76404b585dc6d7, State: Initialized, Role: FOLLOWER
I20260929 18:42:33.475451 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:33.467283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58778 (local address 127.7.157.126:39513)
0929 18:42:33.467957 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:33.467972 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:33.468008 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:42:33.468686 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:33.468705 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:33.469147 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:33.469497 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:33.469912 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.470871 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.472110 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.472595 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.472719 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:33.473546 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:33.473551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:33.473565 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:42:33.473979 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:33.474038 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:42:33.474059 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:33.474132 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:33.474288 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 18:42:33.474569 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":142,"threads_started":1}
I20260929 18:42:33.475736 13105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cad64f1a14973af76404b585dc6d7" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39513 } }
I20260929 18:42:33.475867 13105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:33.475929 13105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:33.476032 13105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:33.476919 13105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cad64f1a14973af76404b585dc6d7" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39513 } }
I20260929 18:42:33.477294 13105 leader_election.cc:302] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 881cad64f1a14973af76404b585dc6d7; no voters:
I20260929 18:42:33.477666 13105 leader_election.cc:288] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:33.477753 13110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:33.477921 13110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [term 1 LEADER]: Becoming Leader. State: Replica: 881cad64f1a14973af76404b585dc6d7, State: Running, Role: LEADER
I20260929 18:42:33.478240 13110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cad64f1a14973af76404b585dc6d7" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39513 } }
I20260929 18:42:33.478350 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:42:33.479027 13105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:33.479892 13112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "881cad64f1a14973af76404b585dc6d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cad64f1a14973af76404b585dc6d7" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39513 } } }
I20260929 18:42:33.480031 13112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:33.480334 13111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 881cad64f1a14973af76404b585dc6d7. Latest consensus state: current_term: 1 leader_uuid: "881cad64f1a14973af76404b585dc6d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cad64f1a14973af76404b585dc6d7" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39513 } } }
I20260929 18:42:33.480423 13111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:33.481382 13119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:33.484586 13119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:33.490731 13119 catalog_manager.cc:1347] Generated new cluster ID: 041f149a9e304660b8ae8a6a68f1038d
I20260929 18:42:33.490808 13119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:33.505160 13119 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:33.506423 13119 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:33.517310 13119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 881cad64f1a14973af76404b585dc6d7: Generated new TSK 0
I20260929 18:42:33.518149 13119 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 18:42:33.544482 13135 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 18:42:33.544577 13135 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 18:42:33.545193 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:33.529396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.7.157.126:39513)
0929 18:42:33.529520 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:33.529524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:33.529635 (+ 111us) server_negotiation.cc:407] Connection header received
0929 18:42:33.529796 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:33.529798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:33.529844 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:33.529932 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:42:33.530150 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.530818 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.532102 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.532333 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.532413 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:33.532700 (+ 287us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:42:33.544833 (+ 12133us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45023/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 18:42:33.545060 (+ 227us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58792: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45023/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 18:42:33.545254 13108 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58792: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45023/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 18:42:33.545429 13134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:33.529245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39513 (local address 127.0.0.1:58792)
0929 18:42:33.529546 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:33.529562 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:33.529680 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:33.529944 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:33.529947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:33.529954 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:42:33.530066 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:33.530072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.530946 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.530950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:33.531986 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:33.531997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.532436 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.532440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:33.532525 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:33.532553 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:42:33.545029 (+ 12476us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45023/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 18:42:33.545242 (+ 213us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.157.126:39513: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45023/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":116,"threads_started":1}
I20260929 18:42:33.545974 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 13038
2026-09-29T18:42:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (301 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 18:42:33.583889 7797 mini_oidc.cc:152] Starting JWKS server
I20260929 18:42:33.584736 7797 webserver.cc:460] Webserver started at https://127.0.0.1:45423/ using document root <none> and password file <none>
I20260929 18:42:33.584822 7797 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:42:33.585054 7797 webserver.cc:460] Webserver started at http://127.0.0.1:42049/ using document root <none> and password file <none>
2026-09-29T18:42:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:33Z Disabled control of system clock
I20260929 18:42:33.593340 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:43409
--webserver_interface=127.7.157.126
--webserver_port=0
--builtin_ntp_servers=127.7.157.84:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:43409
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45423/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 18:42:33.702601 13141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:33.702883 13141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:33.702921 13141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:33.706871 13141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:33.706930 13141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:33.706944 13141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:33.706960 13141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:33.706987 13141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:42:33.707016 13141 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:42:33.707041 13141 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45423/jwks.json
I20260929 18:42:33.711851 13141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:43409
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:43409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45423/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:33.712936 13141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:33.714344 13141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:33.722827 13149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:33.722887 13146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:33.722869 13147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:33.722924 13141 server_base.cc:1034] running on GCE node
I20260929 18:42:33.723594 13141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:33.724704 13141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:33.725857 13141 hybrid_clock.cc:648] HybridClock initialized: now 1790707353725831 us; error 40 us; skew 500 ppm
I20260929 18:42:33.727694 13141 webserver.cc:460] Webserver started at http://127.7.157.126:41977/ using document root <none> and password file <none>
I20260929 18:42:33.728296 13141 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:33.728361 13141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:33.728559 13141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:33.730602 13141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "8f70e466340844e39485c87e1cb301d9"
format_stamp: "Formatted at 2026-09-29 18:42:33 on dist-test-slave-smpn"
server_key: "07963936e489f9aab1b3cd077d526c6e"
server_key_iv: "437c7a16e76009d6bb3ca00f31ef593f"
server_key_version: "encryptionkey@0"
I20260929 18:42:33.731171 13141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "8f70e466340844e39485c87e1cb301d9"
format_stamp: "Formatted at 2026-09-29 18:42:33 on dist-test-slave-smpn"
server_key: "07963936e489f9aab1b3cd077d526c6e"
server_key_iv: "437c7a16e76009d6bb3ca00f31ef593f"
server_key_version: "encryptionkey@0"
I20260929 18:42:33.734813 13141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:42:33.737147 13155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:33.738387 13141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:42:33.738497 13141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "8f70e466340844e39485c87e1cb301d9"
format_stamp: "Formatted at 2026-09-29 18:42:33 on dist-test-slave-smpn"
server_key: "07963936e489f9aab1b3cd077d526c6e"
server_key_iv: "437c7a16e76009d6bb3ca00f31ef593f"
server_key_version: "encryptionkey@0"
I20260929 18:42:33.738590 13141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:33.751889 13141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:33.752614 13141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:33.752805 13141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:33.759481 13141 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:43409
I20260929 18:42:33.759496 13207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:43409 every 8 connection(s)
I20260929 18:42:33.760452 13141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:33.763599 13208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:33.768895 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 13141
I20260929 18:42:33.768989 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:33.769176 13208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9: Bootstrap starting.
I20260929 18:42:33.769243 7797 external_mini_cluster.cc:1428] Setting key 2dbc131ccea3d3809b99e72d57784644
I20260929 18:42:33.771795 13208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:33.772656 13208 log.cc:824] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:33.774820 13208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9: No bootstrap required, opened a new log
I20260929 18:42:33.779116 13208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f70e466340844e39485c87e1cb301d9" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 43409 } }
I20260929 18:42:33.779328 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:33.770533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35166 (local address 127.7.157.126:43409)
0929 18:42:33.771241 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:33.771260 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:33.771303 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:42:33.772122 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:33.772147 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:33.772657 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:33.773019 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:33.773381 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.774759 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.775671 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.776364 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.776524 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:33.777243 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:33.777249 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:33.777266 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:42:33.777764 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:33.777834 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:42:33.777862 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:33.777968 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:33.778044 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 18:42:33.778351 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":103,"threads_started":1}
I20260929 18:42:33.779387 13208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:33.779431 13208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f70e466340844e39485c87e1cb301d9, State: Initialized, Role: FOLLOWER
I20260929 18:42:33.779886 13208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f70e466340844e39485c87e1cb301d9" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 43409 } }
I20260929 18:42:33.780038 13208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:33.780099 13208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:33.780190 13208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:33.781122 13208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f70e466340844e39485c87e1cb301d9" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 43409 } }
I20260929 18:42:33.781463 13208 leader_election.cc:302] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f70e466340844e39485c87e1cb301d9; no voters:
I20260929 18:42:33.781788 13208 leader_election.cc:288] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:33.781888 13213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:33.782081 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:42:33.782089 13213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [term 1 LEADER]: Becoming Leader. State: Replica: 8f70e466340844e39485c87e1cb301d9, State: Running, Role: LEADER
I20260929 18:42:33.782420 13213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f70e466340844e39485c87e1cb301d9" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 43409 } }
I20260929 18:42:33.782960 13208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:33.784327 13214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f70e466340844e39485c87e1cb301d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f70e466340844e39485c87e1cb301d9" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 43409 } } }
I20260929 18:42:33.784431 13214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:33.784647 13215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f70e466340844e39485c87e1cb301d9. Latest consensus state: current_term: 1 leader_uuid: "8f70e466340844e39485c87e1cb301d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f70e466340844e39485c87e1cb301d9" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 43409 } } }
I20260929 18:42:33.784770 13215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:33.784876 13224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:33.787739 13224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:33.793403 13224 catalog_manager.cc:1347] Generated new cluster ID: 225033b982bf4511bd00f6efc84a58e6
I20260929 18:42:33.793505 13224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:33.805688 13224 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:33.807008 13224 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:33.824935 13224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f70e466340844e39485c87e1cb301d9: Generated new TSK 0
I20260929 18:42:33.825693 13224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:42:33.846635 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:33.841224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35178 (local address 127.7.157.126:43409)
0929 18:42:33.841344 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:33.841349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:33.841527 (+ 178us) server_negotiation.cc:407] Connection header received
0929 18:42:33.841689 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:33.841691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:33.841742 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:33.841836 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:42:33.842070 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.842697 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.843662 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.843889 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.843977 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:33.844286 (+ 309us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:42:33.846250 (+ 1964us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45423/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 18:42:33.846485 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35178: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45423/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 18:42:33.846714 13211 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35178: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45423/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 18:42:33.848671 13237 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:33.841128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:43409 (local address 127.0.0.1:35178)
0929 18:42:33.841432 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:33.841443 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:33.841554 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:33.841843 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:33.841846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:33.841855 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:42:33.841967 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:33.841974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.842810 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.842814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:33.843549 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:33.843558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:33.844001 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:33.844005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:33.844093 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:33.844116 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:42:33.846450 (+ 2334us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45423/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 18:42:33.848490 (+ 2040us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.157.126:43409: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45423/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":1820,"thread_start_us":120,"threads_started":1}
I20260929 18:42:33.849058 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 13141
E20260929 18:42:33.857492 13238 webserver.cc:533] Webserver: sslize failed:
E20260929 18:42:33.857597 13238 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T18:42:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (303 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T18:42:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:33Z Disabled control of system clock
I20260929 18:42:33.895236 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:35101
--webserver_interface=127.7.157.126
--webserver_port=0
--builtin_ntp_servers=127.7.157.84:46571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:35101
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/testchainca.cert with env {}
W20260929 18:42:34.004539 13242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:34.004823 13242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:34.004863 13242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:34.008833 13242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:34.008893 13242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:34.008908 13242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:34.008922 13242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:34.008936 13242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:42:34.008951 13242 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:42:34.008975 13242 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 18:42:34.014938 13242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:35101
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:35101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:34.016410 13242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:34.017473 13242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:34.026528 13248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:34.026669 13242 server_base.cc:1034] running on GCE node
W20260929 18:42:34.026532 13247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:34.026564 13250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:34.027485 13242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:34.028671 13242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:34.029835 13242 hybrid_clock.cc:648] HybridClock initialized: now 1790707354029815 us; error 33 us; skew 500 ppm
I20260929 18:42:34.040158 13242 webserver.cc:460] Webserver started at http://127.7.157.126:43141/ using document root <none> and password file <none>
I20260929 18:42:34.040804 13242 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:34.040879 13242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:34.041077 13242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:34.043325 13242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "1c1a8fd4bfc646038b805e4ad43eb68c"
format_stamp: "Formatted at 2026-09-29 18:42:34 on dist-test-slave-smpn"
server_key: "2c49983639314c1f4aaee0a1880a9377"
server_key_iv: "2b471e2a49270719a0e8fa9bf2c22e9d"
server_key_version: "encryptionkey@0"
I20260929 18:42:34.043912 13242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "1c1a8fd4bfc646038b805e4ad43eb68c"
format_stamp: "Formatted at 2026-09-29 18:42:34 on dist-test-slave-smpn"
server_key: "2c49983639314c1f4aaee0a1880a9377"
server_key_iv: "2b471e2a49270719a0e8fa9bf2c22e9d"
server_key_version: "encryptionkey@0"
I20260929 18:42:34.047923 13242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:42:34.050513 13256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:34.051813 13242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:34.051918 13242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "1c1a8fd4bfc646038b805e4ad43eb68c"
format_stamp: "Formatted at 2026-09-29 18:42:34 on dist-test-slave-smpn"
server_key: "2c49983639314c1f4aaee0a1880a9377"
server_key_iv: "2b471e2a49270719a0e8fa9bf2c22e9d"
server_key_version: "encryptionkey@0"
I20260929 18:42:34.052014 13242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:34.065702 13242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:34.066371 13242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:34.066521 13242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:34.073099 13242 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:35101
I20260929 18:42:34.073112 13308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:35101 every 8 connection(s)
I20260929 18:42:34.074064 13242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:34.076973 13309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:34.080868 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 13242
I20260929 18:42:34.080968 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:34.081229 7797 external_mini_cluster.cc:1428] Setting key 0663b21c131b66356084ca8ba220b95d
I20260929 18:42:34.082921 13309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c: Bootstrap starting.
I20260929 18:42:34.085202 13309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:34.085944 13309 log.cc:824] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:34.088080 13309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c: No bootstrap required, opened a new log
I20260929 18:42:34.090567 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:34.082604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.7.157.126:35101)
0929 18:42:34.083327 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:34.083344 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:34.083380 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:42:34.084034 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:34.084052 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:34.084459 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:34.084831 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:34.085271 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:34.086327 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:34.087490 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:34.087964 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:34.088083 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:34.088889 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:34.088894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:34.088907 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:42:34.089290 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:34.089342 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:42:34.089361 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:34.089436 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:34.089581 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 18:42:34.089852 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":143,"threads_started":1}
I20260929 18:42:34.091877 13309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1a8fd4bfc646038b805e4ad43eb68c" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35101 } }
I20260929 18:42:34.092144 13309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:34.092201 13309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c1a8fd4bfc646038b805e4ad43eb68c, State: Initialized, Role: FOLLOWER
I20260929 18:42:34.092621 13309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1a8fd4bfc646038b805e4ad43eb68c" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35101 } }
I20260929 18:42:34.092764 13309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:34.092832 13309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:34.092937 13309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:34.093446 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:42:34.093940 13309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1a8fd4bfc646038b805e4ad43eb68c" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35101 } }
I20260929 18:42:34.094280 13309 leader_election.cc:302] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c1a8fd4bfc646038b805e4ad43eb68c; no voters:
I20260929 18:42:34.094619 13309 leader_election.cc:288] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:34.094695 13314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:34.094897 13314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [term 1 LEADER]: Becoming Leader. State: Replica: 1c1a8fd4bfc646038b805e4ad43eb68c, State: Running, Role: LEADER
I20260929 18:42:34.095237 13314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1a8fd4bfc646038b805e4ad43eb68c" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35101 } }
I20260929 18:42:34.095840 13309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:34.096876 13316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c1a8fd4bfc646038b805e4ad43eb68c. Latest consensus state: current_term: 1 leader_uuid: "1c1a8fd4bfc646038b805e4ad43eb68c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1a8fd4bfc646038b805e4ad43eb68c" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35101 } } }
I20260929 18:42:34.096887 13315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c1a8fd4bfc646038b805e4ad43eb68c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1a8fd4bfc646038b805e4ad43eb68c" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35101 } } }
I20260929 18:42:34.097009 13315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:34.097009 13316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:34.097410 13323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:34.100703 13323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:34.106000 13323 catalog_manager.cc:1347] Generated new cluster ID: 130829a2e4e9475b8a81a20974dd2658
I20260929 18:42:34.106081 13323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:34.125707 13323 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:34.126657 13323 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:34.134603 13323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c1a8fd4bfc646038b805e4ad43eb68c: Generated new TSK 0
I20260929 18:42:34.135411 13323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:42:34.620877 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:34.148757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43260 (local address 127.7.157.126:35101)
0929 18:42:34.148897 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:34.148901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:34.149189 (+ 288us) server_negotiation.cc:407] Connection header received
0929 18:42:34.149628 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:34.149632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:34.149691 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:34.149801 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:42:34.150113 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:34.150958 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:34.152224 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:34.152603 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:34.152720 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:34.155286 (+ 2566us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:42:34.620468 (+465182us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 18:42:34.620751 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43260: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 18:42:34.620962 13312 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43260: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 18:42:34.621449 13338 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:34.148650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:35101 (local address 127.0.0.1:43260)
0929 18:42:34.149097 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:34.149111 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:34.149279 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:34.149831 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:34.149835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:34.149844 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:42:34.149966 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:34.149973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:34.151120 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:34.151124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:34.152076 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:34.152087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:34.152758 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:34.152762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:34.152866 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:34.152896 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:42:34.620728 (+467832us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 18:42:34.620942 (+ 214us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.157.126:35101: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":111,"threads_started":1}
I20260929 18:42:34.622041 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 13242
2026-09-29T18:42:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (752 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (122 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (123 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:42:35 dist-test-slave-smpn krb5kdc[13350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:42:35 dist-test-slave-smpn krb5kdc[13350](info): set up 2 sockets
Sep 29 18:42:35 dist-test-slave-smpn krb5kdc[13350](info): commencing operation
krb5kdc: starting...
W20260929 18:42:36.675150 7797 mini_kdc.cc:121] Time spent starting KDC: real 1.788s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:42:36 dist-test-slave-smpn krb5kdc[13350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:42:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:42:36.793735 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:46337
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:46337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:36.905242 13363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:36.905540 13363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:36.905586 13363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:36.909672 13363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:36.909736 13363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:36.909755 13363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:36.909770 13363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:36.909786 13363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:36.915169 13363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:46337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:36.916298 13363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:36.917342 13363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:36.925487 13368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:36.925487 13371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:36.925586 13363 server_base.cc:1034] running on GCE node
W20260929 18:42:36.925506 13369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:36.926265 13363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:36.927412 13363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:36.928589 13363 hybrid_clock.cc:648] HybridClock initialized: now 1790707356928573 us; error 26 us; skew 500 ppm
Sep 29 18:42:36 dist-test-slave-smpn krb5kdc[13350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707356, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:36.934037 13363 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:42:36.934899 13363 webserver.cc:460] Webserver started at http://127.7.157.126:41847/ using document root <none> and password file <none>
I20260929 18:42:36.935467 13363 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:36.935535 13363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:36.935724 13363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:36.937708 13363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "36e241348da748b28b246c8bc5b251fa"
format_stamp: "Formatted at 2026-09-29 18:42:36 on dist-test-slave-smpn"
server_key: "d9199f2f9378f26376ad28c5feed307a"
server_key_iv: "986e69db80b81bba85faf96affe5242c"
server_key_version: "encryptionkey@0"
I20260929 18:42:36.938231 13363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "36e241348da748b28b246c8bc5b251fa"
format_stamp: "Formatted at 2026-09-29 18:42:36 on dist-test-slave-smpn"
server_key: "d9199f2f9378f26376ad28c5feed307a"
server_key_iv: "986e69db80b81bba85faf96affe5242c"
server_key_version: "encryptionkey@0"
I20260929 18:42:36.941705 13363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 18:42:36.943935 13378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:36.945061 13363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:36.945168 13363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "36e241348da748b28b246c8bc5b251fa"
format_stamp: "Formatted at 2026-09-29 18:42:36 on dist-test-slave-smpn"
server_key: "d9199f2f9378f26376ad28c5feed307a"
server_key_iv: "986e69db80b81bba85faf96affe5242c"
server_key_version: "encryptionkey@0"
I20260929 18:42:36.945256 13363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:36.970920 13363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:36.974152 13363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:36.974354 13363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:36.981492 13363 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:46337
I20260929 18:42:36.981490 13430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:46337 every 8 connection(s)
I20260929 18:42:36.982496 13363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:36.985317 13431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:36.989632 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 13363
I20260929 18:42:36.989740 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:36.989972 7797 external_mini_cluster.cc:1428] Setting key f333b505b952d8495c8702efd4c71a50
I20260929 18:42:36.990936 13431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa: Bootstrap starting.
I20260929 18:42:36.993531 13431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:36.994341 13431 log.cc:824] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:36.996297 13431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa: No bootstrap required, opened a new log
Sep 29 18:42:36 dist-test-slave-smpn krb5kdc[13350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:37.001137 13431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e241348da748b28b246c8bc5b251fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46337 } }
I20260929 18:42:37.001361 13431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:37.001395 13431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e241348da748b28b246c8bc5b251fa, State: Initialized, Role: FOLLOWER
I20260929 18:42:37.001804 13431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e241348da748b28b246c8bc5b251fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46337 } }
I20260929 18:42:37.001925 13431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:37.001988 13431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:37.002079 13431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:37.003206 13431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e241348da748b28b246c8bc5b251fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46337 } }
I20260929 18:42:37.003580 13431 leader_election.cc:302] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36e241348da748b28b246c8bc5b251fa; no voters:
I20260929 18:42:37.003904 13431 leader_election.cc:288] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:37.003978 13436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:37.004166 13436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [term 1 LEADER]: Becoming Leader. State: Replica: 36e241348da748b28b246c8bc5b251fa, State: Running, Role: LEADER
I20260929 18:42:37.004474 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:36.991329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.7.157.126:46337)
0929 18:42:36.991739 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:36.991756 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:36.991797 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:42:36.992603 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:36.992625 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:36.993033 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:36.993384 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:36.994070 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:36.995348 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:36.996161 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:36.996818 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:36.996967 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:36.999717 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:36.999722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:36.999735 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:36.999762 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:37.001354 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:37.001806 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:37.001812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:37.001817 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:37.001929 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:37.002225 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:37.002229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:37.002231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:37.002739 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:37.002937 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:37.003168 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 18:42:37.003483 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":106,"threads_started":1}
I20260929 18:42:37.004499 13436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e241348da748b28b246c8bc5b251fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46337 } }
I20260929 18:42:37.005095 13431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:37.007272 13437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36e241348da748b28b246c8bc5b251fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e241348da748b28b246c8bc5b251fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46337 } } }
I20260929 18:42:37.007628 13437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:37.007555 13438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36e241348da748b28b246c8bc5b251fa. Latest consensus state: current_term: 1 leader_uuid: "36e241348da748b28b246c8bc5b251fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e241348da748b28b246c8bc5b251fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46337 } } }
I20260929 18:42:37.007900 13438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa [sys.catalog]: This master's current role is: LEADER
W20260929 18:42:37.009832 13450 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:42:37.009908 13450 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:42:37.009994 13451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:37.010826 13451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:37.015981 13451 catalog_manager.cc:1347] Generated new cluster ID: 09b0e995275c4467849063e5971b0573
I20260929 18:42:37.016052 13451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:37.037814 13451 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:37.039187 13451 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:37.052510 13451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36e241348da748b28b246c8bc5b251fa: Generated new TSK 0
I20260929 18:42:37.053380 13451 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:42:37.079636 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:46337
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:37.189092 13457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:37.189369 13457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:37.189412 13457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:37.193545 13457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:37.193614 13457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:37.193687 13457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:42:37.199055 13457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:37.200191 13457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:37.201239 13457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:37.210690 13465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:37.210676 13462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:37.210706 13463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:37.210851 13457 server_base.cc:1034] running on GCE node
I20260929 18:42:37.211349 13457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:37.211980 13457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:37.213140 13457 hybrid_clock.cc:648] HybridClock initialized: now 1790707357213118 us; error 34 us; skew 500 ppm
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707357, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:37.215878 13457 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:42:37.216744 13457 webserver.cc:460] Webserver started at http://127.7.157.65:36209/ using document root <none> and password file <none>
I20260929 18:42:37.217263 13457 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:37.217319 13457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:37.217482 13457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:37.219571 13457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "be453acbe6c143759dea7fbdc404d1fb"
format_stamp: "Formatted at 2026-09-29 18:42:37 on dist-test-slave-smpn"
server_key: "9650a45bf3a8c26e4bc3a8fcb9e3c36d"
server_key_iv: "f7245dac9fbd5d14ddd3a7f5c2092ed2"
server_key_version: "encryptionkey@0"
I20260929 18:42:37.220101 13457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "be453acbe6c143759dea7fbdc404d1fb"
format_stamp: "Formatted at 2026-09-29 18:42:37 on dist-test-slave-smpn"
server_key: "9650a45bf3a8c26e4bc3a8fcb9e3c36d"
server_key_iv: "f7245dac9fbd5d14ddd3a7f5c2092ed2"
server_key_version: "encryptionkey@0"
I20260929 18:42:37.223809 13457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:42:37.226228 13472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:37.227497 13457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:37.227600 13457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "be453acbe6c143759dea7fbdc404d1fb"
format_stamp: "Formatted at 2026-09-29 18:42:37 on dist-test-slave-smpn"
server_key: "9650a45bf3a8c26e4bc3a8fcb9e3c36d"
server_key_iv: "f7245dac9fbd5d14ddd3a7f5c2092ed2"
server_key_version: "encryptionkey@0"
I20260929 18:42:37.227715 13457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:37.239212 13457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:37.242388 13457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:37.242569 13457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:37.243196 13457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:37.244028 13457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:37.244076 13457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:37.244118 13457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:37.244146 13457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:37.252948 13457 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:36659
I20260929 18:42:37.254218 13457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:42:37.257357 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 13457
I20260929 18:42:37.257481 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:42:37.257792 7797 external_mini_cluster.cc:1428] Setting key bc7a8e71d982e84461e982d693c9e947
I20260929 18:42:37.254796 13585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:36659 every 8 connection(s)
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707357, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
I20260929 18:42:37.276278 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:37.262748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:32839 (local address 127.7.157.126:46337)
0929 18:42:37.262870 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:37.262875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:37.262889 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:42:37.262932 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:37.262935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:37.262983 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:37.263060 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:37.264926 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.265741 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.267564 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.267938 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.268023 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:37.271611 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:37.271615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:37.271618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:37.271651 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:37.273567 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:37.274306 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:37.274310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:37.274312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:37.274412 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:37.274779 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:37.274782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:37.274784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:37.275025 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:37.275132 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:37.275952 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 18:42:37.276089 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:42:37.277345 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:37.259404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:46337 (local address 127.7.157.65:32839)
0929 18:42:37.260533 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:37.260597 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:37.262325 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:37.263426 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:37.263436 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:37.263981 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:37.264736 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:37.264761 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.265918 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.265921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:37.267415 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:37.267429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.268252 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.268257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:37.268477 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:37.269211 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:37.269233 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:37.271380 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:37.273726 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:37.273733 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:37.273753 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:37.274148 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:37.274519 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:37.274521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:37.274523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:37.274663 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:37.275195 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:37.275203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:37.275625 (+ 422us) client_negotiation.cc:769] Sending connection context
0929 18:42:37.275877 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 18:42:37.276256 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"thread_start_us":140,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
I20260929 18:42:37.278671 13586 heartbeater.cc:347] Connected to a master server at 127.7.157.126:46337
I20260929 18:42:37.278935 13586 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:37.279560 13586 heartbeater.cc:511] Master 127.7.157.126:46337 requested a full tablet report, sending...
I20260929 18:42:37.281061 13395 ts_manager.cc:194] Registered new tserver with Master: be453acbe6c143759dea7fbdc404d1fb (127.7.157.65:36659)
I20260929 18:42:37.282362 13395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:32839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:42:37.293803 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:46337
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:37.406284 13591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:37.406574 13591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:37.406623 13591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:37.410724 13591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:37.410789 13591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:37.410846 13591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:42:37.416100 13591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:37.417140 13591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:37.418131 13591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:37.427729 13596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:37.427765 13597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:37.427765 13599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:37.427968 13591 server_base.cc:1034] running on GCE node
I20260929 18:42:37.428478 13591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:37.429174 13591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:37.430337 13591 hybrid_clock.cc:648] HybridClock initialized: now 1790707357430314 us; error 37 us; skew 500 ppm
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707357, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:37.433054 13591 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:42:37.433976 13591 webserver.cc:460] Webserver started at http://127.7.157.66:40929/ using document root <none> and password file <none>
I20260929 18:42:37.434526 13591 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:37.434592 13591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:37.434777 13591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:37.436859 13591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "7f6ece4eafe942a8b02a04e88a8230cc"
format_stamp: "Formatted at 2026-09-29 18:42:37 on dist-test-slave-smpn"
server_key: "a55a59474c4b1fe77e920742ad6404e3"
server_key_iv: "71fc4679ecb3f0749345c4f3e98206d0"
server_key_version: "encryptionkey@0"
I20260929 18:42:37.437398 13591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "7f6ece4eafe942a8b02a04e88a8230cc"
format_stamp: "Formatted at 2026-09-29 18:42:37 on dist-test-slave-smpn"
server_key: "a55a59474c4b1fe77e920742ad6404e3"
server_key_iv: "71fc4679ecb3f0749345c4f3e98206d0"
server_key_version: "encryptionkey@0"
I20260929 18:42:37.441026 13591 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:42:37.443365 13606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:37.444542 13591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:42:37.444645 13591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "7f6ece4eafe942a8b02a04e88a8230cc"
format_stamp: "Formatted at 2026-09-29 18:42:37 on dist-test-slave-smpn"
server_key: "a55a59474c4b1fe77e920742ad6404e3"
server_key_iv: "71fc4679ecb3f0749345c4f3e98206d0"
server_key_version: "encryptionkey@0"
I20260929 18:42:37.444736 13591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:37.475037 13591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:37.477998 13591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:37.478185 13591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:37.478811 13591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:37.479699 13591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:37.479748 13591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:37.479801 13591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:37.479827 13591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:37.488014 13591 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:46109
I20260929 18:42:37.488039 13719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:46109 every 8 connection(s)
I20260929 18:42:37.488901 13591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:42:37.489972 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 13591
I20260929 18:42:37.490075 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:42:37.490365 7797 external_mini_cluster.cc:1428] Setting key 8f70736d666135cd54b82d68874e2ec9
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707357, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
I20260929 18:42:37.503362 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:37.490793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:36511 (local address 127.7.157.126:46337)
0929 18:42:37.490938 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:37.490943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:37.491876 (+ 933us) server_negotiation.cc:407] Connection header received
0929 18:42:37.493244 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:37.493248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:37.493302 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:37.493390 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:37.494549 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.495100 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.496879 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.497097 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.497169 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:37.499993 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:37.499997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:37.499999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:37.500025 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:37.501266 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:37.501798 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:37.501801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:37.501802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:37.501889 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:37.502214 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:37.502226 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:37.502227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:37.502419 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:37.502495 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:37.503036 (+ 541us) server_negotiation.cc:299] Negotiation successful
0929 18:42:37.503198 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:42:37.504417 13722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:37.491246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:46337 (local address 127.7.157.66:36511)
0929 18:42:37.491723 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:37.491774 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:37.493031 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:37.493539 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:37.493546 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:37.493912 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:37.494401 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:37.494415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.495258 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.495263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:37.496760 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:37.496773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.497261 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.497265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:37.497480 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:37.498083 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:37.498103 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:37.499820 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:37.501365 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:37.501370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:37.501384 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:37.501716 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:37.501961 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:37.501963 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:37.501964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:37.502136 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:37.502515 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:37.502522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:37.502793 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 18:42:37.502959 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 18:42:37.503279 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":110,"threads_started":1}
I20260929 18:42:37.505748 13720 heartbeater.cc:347] Connected to a master server at 127.7.157.126:46337
I20260929 18:42:37.506033 13720 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:37.506644 13720 heartbeater.cc:511] Master 127.7.157.126:46337 requested a full tablet report, sending...
I20260929 18:42:37.507876 13395 ts_manager.cc:194] Registered new tserver with Master: 7f6ece4eafe942a8b02a04e88a8230cc (127.7.157.66:46109)
I20260929 18:42:37.508489 13395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:36511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:42:37.518548 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:46337
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:37.626135 13725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:37.626415 13725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:37.626474 13725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:37.630541 13725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:37.630614 13725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:37.630695 13725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:42:37.636099 13725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:37.637181 13725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:37.638227 13725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:37.647909 13733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:37.647886 13730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:37.647886 13731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:37.647940 13725 server_base.cc:1034] running on GCE node
I20260929 18:42:37.648579 13725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:37.649303 13725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:37.650471 13725 hybrid_clock.cc:648] HybridClock initialized: now 1790707357650449 us; error 34 us; skew 500 ppm
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707357, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:37.653393 13725 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:42:37.654366 13725 webserver.cc:460] Webserver started at http://127.7.157.67:41351/ using document root <none> and password file <none>
I20260929 18:42:37.654934 13725 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:37.655000 13725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:37.655249 13725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:37.657411 13725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "5719e60bb5524000824031c47e37df03"
format_stamp: "Formatted at 2026-09-29 18:42:37 on dist-test-slave-smpn"
server_key: "9389f8bfd1c1807ed481d80ad0b0cfda"
server_key_iv: "cef29f7357126b7352a88a03683a2451"
server_key_version: "encryptionkey@0"
I20260929 18:42:37.657975 13725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "5719e60bb5524000824031c47e37df03"
format_stamp: "Formatted at 2026-09-29 18:42:37 on dist-test-slave-smpn"
server_key: "9389f8bfd1c1807ed481d80ad0b0cfda"
server_key_iv: "cef29f7357126b7352a88a03683a2451"
server_key_version: "encryptionkey@0"
I20260929 18:42:37.661783 13725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:42:37.664379 13740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:37.665676 13725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:37.665795 13725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "5719e60bb5524000824031c47e37df03"
format_stamp: "Formatted at 2026-09-29 18:42:37 on dist-test-slave-smpn"
server_key: "9389f8bfd1c1807ed481d80ad0b0cfda"
server_key_iv: "cef29f7357126b7352a88a03683a2451"
server_key_version: "encryptionkey@0"
I20260929 18:42:37.665901 13725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:37.681607 13725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:37.684679 13725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:37.684849 13725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:37.685444 13725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:37.686250 13725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:37.686296 13725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:37.686347 13725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:37.686375 13725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:37.694938 13725 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:37813
I20260929 18:42:37.695011 13853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:37813 every 8 connection(s)
I20260929 18:42:37.695851 13725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:42:37.704005 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 13725
I20260929 18:42:37.704103 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:42:37.704331 7797 external_mini_cluster.cc:1428] Setting key b9a3d295fbebaa54feabf220fa9ae5f0
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707357, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:37.708968 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:37.697572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:47747 (local address 127.7.157.126:46337)
0929 18:42:37.697704 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:37.697708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:37.698531 (+ 823us) server_negotiation.cc:407] Connection header received
0929 18:42:37.699934 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:37.699938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:37.699996 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:37.700072 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:37.701269 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.701805 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.702950 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.703224 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.703319 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:37.705876 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:37.705879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:37.705881 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:37.705906 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:37.707272 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:37.707749 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:37.707751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:37.707753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:37.707823 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:37.708079 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:37.708081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:37.708082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:37.708250 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:37.708318 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:37.708752 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 18:42:37.708848 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:42:37.709867 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:37.697951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:46337 (local address 127.7.157.67:47747)
0929 18:42:37.698390 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:37.698432 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:37.699712 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:37.700267 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:37.700275 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:37.700630 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:37.701121 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:37.701134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.701922 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.701925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:37.702830 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:37.702839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.703373 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.703377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:37.703536 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:37.704124 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:37.704145 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:37.705709 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:37.707382 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:37.707387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:37.707401 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:37.707658 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:37.707885 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:37.707887 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:37.707889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:37.708018 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:37.708319 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:37.708324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:37.708561 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 18:42:37.708730 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 18:42:37.708994 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":107,"threads_started":1}
I20260929 18:42:37.711014 13854 heartbeater.cc:347] Connected to a master server at 127.7.157.126:46337
I20260929 18:42:37.711273 13854 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:37.711738 13854 heartbeater.cc:511] Master 127.7.157.126:46337 requested a full tablet report, sending...
I20260929 18:42:37.712705 13395 ts_manager.cc:194] Registered new tserver with Master: 5719e60bb5524000824031c47e37df03 (127.7.157.67:37813)
I20260929 18:42:37.713238 13395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:47747
I20260929 18:42:37.718077 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/test-admin.keytab.
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:37.731411 7797 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 18:42:37.733958 13861 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:37.732654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.7.157.65:36659)
0929 18:42:37.732962 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:37.732977 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:37.733022 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:42:37.733071 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:37.733077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:37.733280 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:37.733390 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:37.733788 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43638: BlockingRecv error: recv got EOF from 127.0.0.1:43638 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":125,"threads_started":1}
W20260929 18:42:37.734036 7797 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:37.734997 7797 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:37.735880 7797 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.65@KRBTEST.COM
I20260929 18:42:37.744776 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:37.736690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.7.157.65:36659)
0929 18:42:37.736806 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:37.736810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:37.736890 (+ 80us) server_negotiation.cc:407] Connection header received
0929 18:42:37.736995 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:37.736997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:37.737033 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:37.737088 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:37.737525 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.738285 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.739049 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.739477 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.739560 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:37.741583 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:37.741593 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:37.741608 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:37.741631 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:37.743231 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:37.743607 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:37.743612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:37.743613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:37.743688 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:37.743889 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:37.743891 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:37.743892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:37.744129 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:37.744280 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:37.744444 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 18:42:37.744588 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 18:42:37.748674 13861 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:37.747688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.7.157.65:36659)
0929 18:42:37.747810 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:37.747814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:37.747922 (+ 108us) server_negotiation.cc:407] Connection header received
0929 18:42:37.748047 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:37.748049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:37.748086 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:37.748146 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:37.748557 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43664: BlockingRecv error: recv got EOF from 127.0.0.1:43664 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 18:42:37.748740 7797 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:37.749794 7797 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:37.750661 7797 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.65@KRBTEST.COM
I20260929 18:42:37.759408 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:37.751520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.7.157.65:36659)
0929 18:42:37.751638 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:37.751643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:37.751840 (+ 197us) server_negotiation.cc:407] Connection header received
0929 18:42:37.751959 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:37.751962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:37.752004 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:37.752068 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:37.752572 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.753091 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.753886 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.754117 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.754207 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:37.756481 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:37.756484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:37.756486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:37.756514 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:37.757812 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:37.758224 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:37.758228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:37.758230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:37.758308 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:37.758558 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:37.758561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:37.758563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:37.758746 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:37.758860 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:37.759141 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 18:42:37.759255 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 18:42:37.762068 13861 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:37.761075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43676 (local address 127.7.157.65:36659)
0929 18:42:37.761210 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:37.761215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:37.761288 (+ 73us) server_negotiation.cc:407] Connection header received
0929 18:42:37.761414 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:37.761417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:37.761456 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:37.761517 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:37.761966 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43676: BlockingRecv error: recv got EOF from 127.0.0.1:43676 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 18:42:37.762176 7797 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707328618849-7797-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:37.763273 7797 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:37.764122 7797 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.65@KRBTEST.COM
I20260929 18:42:37.771570 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:37.764903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43680 (local address 127.7.157.65:36659)
0929 18:42:37.765012 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:37.765016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:37.765105 (+ 89us) server_negotiation.cc:407] Connection header received
0929 18:42:37.765185 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:37.765187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:37.765222 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:37.765283 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:37.765704 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.766177 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.766904 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:37.767155 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:37.767231 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:37.769096 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:37.769099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:37.769101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:37.769122 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:37.770308 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:37.770645 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:37.770648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:37.770649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:37.770732 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:37.770936 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:37.770938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:37.770939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:37.771118 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:37.771220 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:37.771375 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 18:42:37.771458 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:42:37.772433 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 13457
I20260929 18:42:37.777931 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 13591
I20260929 18:42:37.782334 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 13725
I20260929 18:42:37.786762 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 13363
2026-09-29T18:42:37Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2919 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13874](info): set up 2 sockets
Sep 29 18:42:37 dist-test-slave-smpn krb5kdc[13874](info): commencing operation
krb5kdc: starting...
W20260929 18:42:38.743140 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:42:38 dist-test-slave-smpn krb5kdc[13874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:42:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.126.keytab.
Entry for principal oryx/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.126.keytab.
I20260929 18:42:38.854311 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:45147
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.126.keytab
--principal=oryx/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:45147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:38.964315 13887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:38.964594 13887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:38.964635 13887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:38.968684 13887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:38.968753 13887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:38.968771 13887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:38.968786 13887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:38.968806 13887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:38.974128 13887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:45147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.126.keytab
--principal=oryx/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:45147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:38.975215 13887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:38.976241 13887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:38.984639 13893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:38.984826 13895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:38.984673 13892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:38.984721 13887 server_base.cc:1034] running on GCE node
I20260929 18:42:38.985438 13887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:38.986620 13887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:38.987776 13887 hybrid_clock.cc:648] HybridClock initialized: now 1790707358987759 us; error 34 us; skew 500 ppm
Sep 29 18:42:38 dist-test-slave-smpn krb5kdc[13874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707358, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:38.990429 13887 init.cc:377] Logged in from keytab as oryx/127.7.157.126@KRBTEST.COM (short username oryx)
I20260929 18:42:38.991356 13887 webserver.cc:460] Webserver started at http://127.7.157.126:37923/ using document root <none> and password file <none>
I20260929 18:42:38.991889 13887 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:38.991950 13887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:38.992151 13887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:38.994212 13887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "d3bffeef2ce1474981574d052413f6b6"
format_stamp: "Formatted at 2026-09-29 18:42:38 on dist-test-slave-smpn"
server_key: "8cc6b8672f47f8dca6f437dd3f0c12d7"
server_key_iv: "8b555a083ddc258cc8a5f9984e4b0687"
server_key_version: "encryptionkey@0"
I20260929 18:42:38.994757 13887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "d3bffeef2ce1474981574d052413f6b6"
format_stamp: "Formatted at 2026-09-29 18:42:38 on dist-test-slave-smpn"
server_key: "8cc6b8672f47f8dca6f437dd3f0c12d7"
server_key_iv: "8b555a083ddc258cc8a5f9984e4b0687"
server_key_version: "encryptionkey@0"
I20260929 18:42:38.998430 13887 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:42:39.000785 13902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:39.001959 13887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:42:39.002074 13887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "d3bffeef2ce1474981574d052413f6b6"
format_stamp: "Formatted at 2026-09-29 18:42:38 on dist-test-slave-smpn"
server_key: "8cc6b8672f47f8dca6f437dd3f0c12d7"
server_key_iv: "8b555a083ddc258cc8a5f9984e4b0687"
server_key_version: "encryptionkey@0"
I20260929 18:42:39.002169 13887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:39.025900 13887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:39.028870 13887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:39.029059 13887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:39.035773 13887 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:45147
I20260929 18:42:39.036015 13954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:45147 every 8 connection(s)
I20260929 18:42:39.036742 13887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:39.039498 13955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:39.040512 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 13887
I20260929 18:42:39.040616 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:39.040876 7797 external_mini_cluster.cc:1428] Setting key a6ec924d056dd2f68cde1df7152638fd
I20260929 18:42:39.046304 13955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6: Bootstrap starting.
I20260929 18:42:39.048957 13955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:39.049700 13955 log.cc:824] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:42:39 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM
I20260929 18:42:39.051724 13955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6: No bootstrap required, opened a new log
I20260929 18:42:39.054631 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:39.042172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.7.157.126:45147)
0929 18:42:39.042607 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:39.042628 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:39.042671 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:42:39.043484 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:39.043507 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:39.043933 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:39.044290 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:39.044928 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.046395 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.047253 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.047949 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.048103 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:39.050433 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:39.050439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:39.050453 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:39.050480 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:39.052025 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:39.052432 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:39.052437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:39.052441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:39.052550 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:39.052793 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:39.052796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:39.052798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:39.053181 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:39.053372 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:39.053630 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 18:42:39.053921 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":98,"threads_started":1}
I20260929 18:42:39.055298 13955 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } }
I20260929 18:42:39.055567 13955 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:39.055622 13955 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3bffeef2ce1474981574d052413f6b6, State: Initialized, Role: FOLLOWER
I20260929 18:42:39.056044 13955 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } }
I20260929 18:42:39.056185 13955 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:39.056247 13955 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:39.056339 13955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:39.057338 13955 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } }
I20260929 18:42:39.057650 13955 leader_election.cc:302] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3bffeef2ce1474981574d052413f6b6; no voters:
I20260929 18:42:39.057921 13955 leader_election.cc:288] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:39.058054 13960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:39.058257 13960 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [term 1 LEADER]: Becoming Leader. State: Replica: d3bffeef2ce1474981574d052413f6b6, State: Running, Role: LEADER
I20260929 18:42:39.058595 13960 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } }
I20260929 18:42:39.059106 13955 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:39.060321 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3bffeef2ce1474981574d052413f6b6. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } }
I20260929 18:42:39.060485 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:39.060321 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } }
I20260929 18:42:39.060552 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:39.060927 13970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:39.064162 13970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:39.069767 13970 catalog_manager.cc:1347] Generated new cluster ID: 50298e1848a94311b5e0994750e859f9
I20260929 18:42:39.069834 13970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:39.089578 13970 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:39.090520 13970 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:39.097766 13970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6: Generated new TSK 0
I20260929 18:42:39.098577 13970 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.65.keytab.
Entry for principal oryx/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.65.keytab.
I20260929 18:42:39.134315 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45147
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.65.keytab
--principal=oryx/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:39.244037 13981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:39.244323 13981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:39.244367 13981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:39.248456 13981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:39.248543 13981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:39.248625 13981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:42:39.254127 13981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.65.keytab
--principal=oryx/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:39.255326 13981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:39.256381 13981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:42:39.266135 13981 server_base.cc:1034] running on GCE node
W20260929 18:42:39.266052 13986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:39.266091 13987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:39.266052 13989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:39.266707 13981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:39.267409 13981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:39.268564 13981 hybrid_clock.cc:648] HybridClock initialized: now 1790707359268545 us; error 29 us; skew 500 ppm
Sep 29 18:42:39 dist-test-slave-smpn krb5kdc[13874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707359, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:39.271200 13981 init.cc:377] Logged in from keytab as oryx/127.7.157.65@KRBTEST.COM (short username oryx)
I20260929 18:42:39.272050 13981 webserver.cc:460] Webserver started at http://127.7.157.65:38045/ using document root <none> and password file <none>
I20260929 18:42:39.272553 13981 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:39.272609 13981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:39.272785 13981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:39.274722 13981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "5a8d904fb2c54e72bbf064ec58d2a15a"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "f3005f7f3333f18bede5e37a25975fdd"
server_key_iv: "89a4195fa48c8603a70d937cb7faac33"
server_key_version: "encryptionkey@0"
I20260929 18:42:39.275273 13981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "5a8d904fb2c54e72bbf064ec58d2a15a"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "f3005f7f3333f18bede5e37a25975fdd"
server_key_iv: "89a4195fa48c8603a70d937cb7faac33"
server_key_version: "encryptionkey@0"
I20260929 18:42:39.278795 13981 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:42:39.281239 13996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:39.282519 13981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:39.282646 13981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "5a8d904fb2c54e72bbf064ec58d2a15a"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "f3005f7f3333f18bede5e37a25975fdd"
server_key_iv: "89a4195fa48c8603a70d937cb7faac33"
server_key_version: "encryptionkey@0"
I20260929 18:42:39.282753 13981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:39.317560 13981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:39.320467 13981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:39.320650 13981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:39.321233 13981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:39.322057 13981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:39.322104 13981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:39.322150 13981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:39.322178 13981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:39.330428 13981 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:45901
I20260929 18:42:39.330456 14109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:45901 every 8 connection(s)
I20260929 18:42:39.331370 13981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:42:39 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707359, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM
I20260929 18:42:39.340919 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 13981
I20260929 18:42:39.341046 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:42:39.341317 7797 external_mini_cluster.cc:1428] Setting key d92a75551919dba1c7cfc9500fbd75f7
I20260929 18:42:39.344501 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:39.333179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:52853 (local address 127.7.157.126:45147)
0929 18:42:39.333320 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:39.333323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:39.334163 (+ 840us) server_negotiation.cc:407] Connection header received
0929 18:42:39.335477 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:39.335481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:39.335538 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:39.335612 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:39.336801 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.337331 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.338433 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.338668 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.338757 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:39.341192 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:39.341196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:39.341198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:39.341225 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:39.342535 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:39.343014 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:39.343018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:39.343020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:39.343129 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:39.343450 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:39.343453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:39.343455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:39.343633 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:39.343726 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:39.344222 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 18:42:39.344346 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:42:39.345367 14112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:39.333566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.65:52853)
0929 18:42:39.334003 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:39.334047 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:39.335265 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:39.335802 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:39.335810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:39.336176 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:39.336659 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:39.336673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.337445 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.337448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:39.338320 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:39.338329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.338809 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.338812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:39.338971 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:39.339513 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:39.339528 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:39.341015 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:39.342646 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:39.342652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:39.342669 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:39.342920 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:39.343215 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:39.343219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:39.343221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:39.343363 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:39.343727 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:39.343734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:39.343997 (+ 263us) client_negotiation.cc:769] Sending connection context
0929 18:42:39.344169 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 18:42:39.344514 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":99,"threads_started":1}
I20260929 18:42:39.346545 14110 heartbeater.cc:347] Connected to a master server at 127.7.157.126:45147
I20260929 18:42:39.346786 14110 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:39.347297 14110 heartbeater.cc:511] Master 127.7.157.126:45147 requested a full tablet report, sending...
I20260929 18:42:39.348944 13919 ts_manager.cc:194] Registered new tserver with Master: 5a8d904fb2c54e72bbf064ec58d2a15a (127.7.157.65:45901)
WARNING: no policy specified for oryx/127.7.157.66@KRBTEST.COM; defaulting to no policy
I20260929 18:42:39.350430 13919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:52853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.66.keytab.
Entry for principal oryx/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.66.keytab.
I20260929 18:42:39.365995 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45147
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.66.keytab
--principal=oryx/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:39.475047 14115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:39.475360 14115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:39.475421 14115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:39.479470 14115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:39.479547 14115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:39.479631 14115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:42:39.485054 14115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.66.keytab
--principal=oryx/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:39.486181 14115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:39.487295 14115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:39.497052 14123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:39.497046 14120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:39.497056 14121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:39.497129 14115 server_base.cc:1034] running on GCE node
I20260929 18:42:39.497669 14115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:39.498309 14115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:39.499459 14115 hybrid_clock.cc:648] HybridClock initialized: now 1790707359499427 us; error 41 us; skew 500 ppm
Sep 29 18:42:39 dist-test-slave-smpn krb5kdc[13874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707359, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:39.502008 14115 init.cc:377] Logged in from keytab as oryx/127.7.157.66@KRBTEST.COM (short username oryx)
I20260929 18:42:39.502914 14115 webserver.cc:460] Webserver started at http://127.7.157.66:34275/ using document root <none> and password file <none>
I20260929 18:42:39.503502 14115 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:39.503574 14115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:39.503767 14115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:39.505878 14115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "4ec3bc489671458189f218079f60c6ff"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "a4f31bd3c32c5b13dbd0222b1fa34d0e"
server_key_iv: "7e36442bc5d64c8efb4b32051c3610a7"
server_key_version: "encryptionkey@0"
I20260929 18:42:39.506489 14115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "4ec3bc489671458189f218079f60c6ff"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "a4f31bd3c32c5b13dbd0222b1fa34d0e"
server_key_iv: "7e36442bc5d64c8efb4b32051c3610a7"
server_key_version: "encryptionkey@0"
I20260929 18:42:39.510336 14115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:42:39.512801 14130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:39.514070 14115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:39.514197 14115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "4ec3bc489671458189f218079f60c6ff"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "a4f31bd3c32c5b13dbd0222b1fa34d0e"
server_key_iv: "7e36442bc5d64c8efb4b32051c3610a7"
server_key_version: "encryptionkey@0"
I20260929 18:42:39.514312 14115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:39.534186 14115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:39.537623 14115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:39.537830 14115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:39.538461 14115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:39.539361 14115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:39.539412 14115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:39.539465 14115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:39.539494 14115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:39.547930 14115 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:42467
I20260929 18:42:39.547960 14243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:42467 every 8 connection(s)
I20260929 18:42:39.548832 14115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:42:39.551834 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 14115
I20260929 18:42:39.551955 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:42:39.552264 7797 external_mini_cluster.cc:1428] Setting key 8ed931f9e9067139f1fa080135896724
Sep 29 18:42:39 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707359, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.7.157.67@KRBTEST.COM; defaulting to no policy
I20260929 18:42:39.562999 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:39.550665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:47369 (local address 127.7.157.126:45147)
0929 18:42:39.550809 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:39.550815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:39.551691 (+ 876us) server_negotiation.cc:407] Connection header received
0929 18:42:39.553180 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:39.553184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:39.553228 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:39.553308 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:39.554741 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.555373 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.556499 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.556816 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.556912 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:39.559410 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:39.559414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:39.559417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:39.559447 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:39.561106 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:39.561603 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:39.561607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:39.561610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:39.561703 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:39.561986 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:39.561988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:39.561989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:39.562160 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:39.562245 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:39.562776 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 18:42:39.562876 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.157.67@KRBTEST.COM" created.
I20260929 18:42:39.563897 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:39.551060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.66:47369)
0929 18:42:39.551513 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:39.551571 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:39.552956 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:39.553455 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:39.553463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:39.553914 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:39.554582 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:39.554601 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.555485 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.555488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:39.556387 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:39.556396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.556981 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.556984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:39.557136 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:39.557748 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:39.557765 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:39.559226 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:39.561208 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:39.561213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:39.561225 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:39.561492 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:39.561780 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:39.561782 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:39.561783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:39.561909 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:39.562262 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:39.562267 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:39.562551 (+ 284us) client_negotiation.cc:769] Sending connection context
0929 18:42:39.562717 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 18:42:39.562973 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":101,"threads_started":1}
I20260929 18:42:39.565025 14244 heartbeater.cc:347] Connected to a master server at 127.7.157.126:45147
I20260929 18:42:39.565295 14244 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:39.565771 14244 heartbeater.cc:511] Master 127.7.157.126:45147 requested a full tablet report, sending...
I20260929 18:42:39.566812 13919 ts_manager.cc:194] Registered new tserver with Master: 4ec3bc489671458189f218079f60c6ff (127.7.157.66:42467)
I20260929 18:42:39.567426 13919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:47369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.67.keytab.
Entry for principal oryx/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.67.keytab.
I20260929 18:42:39.578500 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45147
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.67.keytab
--principal=oryx/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:39.685710 14249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:39.686013 14249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:39.686074 14249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:39.690142 14249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:39.690207 14249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:39.690281 14249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:42:39.695591 14249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.67.keytab
--principal=oryx/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:39.696591 14249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:39.697556 14249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:39.706537 14257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:39.706547 14254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:39.706532 14255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:39.706611 14249 server_base.cc:1034] running on GCE node
I20260929 18:42:39.707249 14249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:39.707882 14249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:39.709041 14249 hybrid_clock.cc:648] HybridClock initialized: now 1790707359709018 us; error 35 us; skew 500 ppm
Sep 29 18:42:39 dist-test-slave-smpn krb5kdc[13874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707359, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:39.711550 14249 init.cc:377] Logged in from keytab as oryx/127.7.157.67@KRBTEST.COM (short username oryx)
I20260929 18:42:39.712364 14249 webserver.cc:460] Webserver started at http://127.7.157.67:46613/ using document root <none> and password file <none>
I20260929 18:42:39.712879 14249 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:39.712941 14249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:39.713115 14249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:39.715153 14249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "e28610da940c495f9a268825f9994a3d"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "ba4e8920759584402c4654c47a8c8a59"
server_key_iv: "9087c88e5aa2edfc3be8fca2c8f6ff59"
server_key_version: "encryptionkey@0"
I20260929 18:42:39.715691 14249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "e28610da940c495f9a268825f9994a3d"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "ba4e8920759584402c4654c47a8c8a59"
server_key_iv: "9087c88e5aa2edfc3be8fca2c8f6ff59"
server_key_version: "encryptionkey@0"
I20260929 18:42:39.719334 14249 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:42:39.721589 14264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:39.722760 14249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:39.722864 14249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "e28610da940c495f9a268825f9994a3d"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "ba4e8920759584402c4654c47a8c8a59"
server_key_iv: "9087c88e5aa2edfc3be8fca2c8f6ff59"
server_key_version: "encryptionkey@0"
I20260929 18:42:39.722957 14249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:39.740676 14249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:39.743462 14249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:39.743636 14249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:39.744246 14249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:39.745069 14249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:39.745118 14249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:39.745173 14249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:39.745201 14249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:39.753283 14249 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:37959
I20260929 18:42:39.753350 14377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:37959 every 8 connection(s)
I20260929 18:42:39.754197 14249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:42:39 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707359, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM
I20260929 18:42:39.764287 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 14249
I20260929 18:42:39.764415 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:42:39.764707 7797 external_mini_cluster.cc:1428] Setting key 9064a30a5fbfae6a066c7eee50a6a073
I20260929 18:42:39.767474 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:39.756059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:51261 (local address 127.7.157.126:45147)
0929 18:42:39.756202 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:39.756207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:39.757057 (+ 850us) server_negotiation.cc:407] Connection header received
0929 18:42:39.758275 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:39.758277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:39.758324 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:39.758397 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:39.759674 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.760218 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.761354 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.761571 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.761660 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:39.764101 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:39.764103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:39.764105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:39.764131 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:39.765510 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:39.765996 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:39.766000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:39.766002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:39.766078 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:39.766391 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:39.766394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:39.766395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:39.766580 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:39.766683 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:39.767254 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 18:42:39.767359 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:42:39.768347 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:39.756467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.67:51261)
0929 18:42:39.756902 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:39.756943 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:39.758076 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:39.758557 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:39.758564 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:39.758943 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:39.759510 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:39.759525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.760329 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.760333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:39.761223 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:39.761233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.761722 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.761726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:39.761885 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:39.762449 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:39.762466 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:39.763917 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:39.765617 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:39.765623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:39.765640 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:39.765903 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:39.766156 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:39.766159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:39.766161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:39.766300 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:39.766733 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:39.766740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:39.767000 (+ 260us) client_negotiation.cc:769] Sending connection context
0929 18:42:39.767208 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 18:42:39.767472 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
I20260929 18:42:39.769368 14378 heartbeater.cc:347] Connected to a master server at 127.7.157.126:45147
I20260929 18:42:39.769584 14378 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:39.770059 14378 heartbeater.cc:511] Master 127.7.157.126:45147 requested a full tablet report, sending...
I20260929 18:42:39.770965 13919 ts_manager.cc:194] Registered new tserver with Master: e28610da940c495f9a268825f9994a3d (127.7.157.67:37959)
I20260929 18:42:39.771528 13919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:51261
I20260929 18:42:39.778402 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.157.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.157.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.157.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.125.keytab.
Entry for principal oryx/127.7.157.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.125.keytab.
I20260929 18:42:39.803577 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.125:42879
--webserver_interface=127.7.157.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.125.keytab
--principal=oryx/127.7.157.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:45147,127.7.157.125:42879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:39.912995 14383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:39.913283 14383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:39.913348 14383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:39.917430 14383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:39.917507 14383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:39.917529 14383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:39.917546 14383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:39.917565 14383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:39.923013 14383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:45147,127.7.157.125:42879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.125.keytab
--principal=oryx/127.7.157.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.125:42879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.157.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
W20260929 18:42:39.923271 14383 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.7.157.126:45147,127.7.157.125:42879'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 18:42:39.924108 14383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:39.925040 14383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:39.933007 14391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:39.933014 14388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:39.933015 14389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:39.933159 14383 server_base.cc:1034] running on GCE node
I20260929 18:42:39.933799 14383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:39.934813 14383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:39.935959 14383 hybrid_clock.cc:648] HybridClock initialized: now 1790707359935927 us; error 48 us; skew 500 ppm
Sep 29 18:42:39 dist-test-slave-smpn krb5kdc[13874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707359, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:39.938493 14383 init.cc:377] Logged in from keytab as oryx/127.7.157.125@KRBTEST.COM (short username oryx)
I20260929 18:42:39.939352 14383 webserver.cc:460] Webserver started at http://127.7.157.125:42293/ using document root <none> and password file <none>
I20260929 18:42:39.939890 14383 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:39.939949 14383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:39.940140 14383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:39.942075 14383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/data/instance:
uuid: "e697954f7ed54a67a0a9a51adfc5e1ba"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "2413ed0cc59f0d3840ffc20e3c703c83"
server_key_iv: "c36c91c7a5beaee353ed452ea58877c6"
server_key_version: "encryptionkey@0"
I20260929 18:42:39.942595 14383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/wal/instance:
uuid: "e697954f7ed54a67a0a9a51adfc5e1ba"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "2413ed0cc59f0d3840ffc20e3c703c83"
server_key_iv: "c36c91c7a5beaee353ed452ea58877c6"
server_key_version: "encryptionkey@0"
I20260929 18:42:39.946062 14383 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:42:39.948246 14398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:39.949379 14383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:39.949477 14383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/wal
uuid: "e697954f7ed54a67a0a9a51adfc5e1ba"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "2413ed0cc59f0d3840ffc20e3c703c83"
server_key_iv: "c36c91c7a5beaee353ed452ea58877c6"
server_key_version: "encryptionkey@0"
I20260929 18:42:39.949565 14383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:39.962384 14383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:39.965139 14383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:39.965308 14383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:39.971848 14383 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.125:42879
I20260929 18:42:39.972086 14450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.125:42879 every 8 connection(s)
I20260929 18:42:39.972788 14383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/data/info.pb
I20260929 18:42:39.975514 14451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:39.978965 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 14383
I20260929 18:42:39.979177 14451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:42:39 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707359, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.125@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM
Sep 29 18:42:39 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.157.125@KRBTEST.COM
I20260929 18:42:39.991043 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:39.981158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.7.157.126:45147)
0929 18:42:39.981278 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:39.981282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:39.981508 (+ 226us) server_negotiation.cc:407] Connection header received
0929 18:42:39.981853 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:39.981856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:39.981907 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:39.981976 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:39.983367 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.983939 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.985152 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.985400 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.985494 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:39.987673 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:39.987677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:39.987680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:39.987707 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:39.989092 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:39.989638 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:39.989641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:39.989643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:39.989716 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:39.990007 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:39.990010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:39.990012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:39.990229 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:39.990313 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:39.990770 (+ 457us) server_negotiation.cc:299] Negotiation successful
0929 18:42:39.990882 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:42:39.991899 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:39.981084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:51910)
0929 18:42:39.981392 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:39.981429 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:39.981693 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:39.982741 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:39.982745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:39.983050 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:39.983245 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:39.983255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.984058 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.984061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:39.985041 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:39.985051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.985536 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.985540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:39.985647 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:39.986113 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:39.986130 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:39.987539 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:39.989184 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:39.989192 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:39.989202 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:39.989552 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:39.989795 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:39.989798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:39.989800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:39.989925 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:39.990326 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:39.990332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:39.990569 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 18:42:39.990723 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 18:42:39.991034 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":55,"spinlock_wait_cycles":24192,"thread_start_us":79,"threads_started":1}
I20260929 18:42:39.991904 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:39.980410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.7.157.125:42879)
0929 18:42:39.980792 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:39.980810 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:39.980847 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:42:39.981645 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:39.981666 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:39.981872 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:39.982009 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:39.982694 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.983678 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.984592 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.985259 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.985369 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:39.988072 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:39.988077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:39.988087 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:39.988106 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:39.989536 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:39.989918 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:39.989924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:39.989926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:39.989992 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:39.990236 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:39.990239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:39.990241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:39.990559 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:39.990731 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:39.990932 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 18:42:39.991119 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
I20260929 18:42:39.993292 14451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:42:39 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707359, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.125@KRBTEST.COM for oryx/127.7.157.125@KRBTEST.COM
I20260929 18:42:40.000465 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:39.993822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.7.157.125:42879)
0929 18:42:39.993918 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:39.993922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:39.993993 (+ 71us) server_negotiation.cc:407] Connection header received
0929 18:42:39.994080 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:39.994083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:39.994117 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:39.994171 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:39.994493 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.994960 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.995670 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.996296 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.996382 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:39.997945 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:39.997948 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:39.997949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:39.997967 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:39.998976 (+ 1009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:39.999334 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:39.999337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:39.999339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:39.999400 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:39.999611 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:39.999614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:39.999616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:39.999875 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:39.999983 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.000210 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.000341 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":33}
I20260929 18:42:40.000465 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:39.993753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42879 (local address 127.0.0.1:37398)
0929 18:42:39.993915 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:39.993934 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:39.994018 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:39.994190 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:39.994193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:39.994331 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:39.994403 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:39.994408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.995047 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.995050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:39.995577 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:39.995584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:39.996388 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:39.996392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:39.996480 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:39.996823 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:39.996833 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:39.997837 (+ 1004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:39.999085 (+ 1248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:39.999088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:39.999090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:39.999256 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:39.999473 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:39.999475 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:39.999477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:39.999544 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:39.999987 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:39.999990 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.000079 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.000161 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.000292 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 18:42:40.003196 14451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: Bootstrap starting.
I20260929 18:42:40.005429 14451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:40.006174 14451 log.cc:824] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:40.008247 14451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: No bootstrap required, opened a new log
I20260929 18:42:40.011891 14451 raft_consensus.cc:348] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } }
I20260929 18:42:40.012089 14451 raft_consensus.cc:374] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:40.012140 14451 raft_consensus.cc:729] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e697954f7ed54a67a0a9a51adfc5e1ba, State: Initialized, Role: FOLLOWER
I20260929 18:42:40.012571 14451 consensus_queue.cc:260] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } }
I20260929 18:42:40.013029 14457 sys_catalog.cc:455] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } } }
I20260929 18:42:40.013145 14457 sys_catalog.cc:458] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:42:40.013715 14451 sys_catalog.cc:564] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:42:40.018280 14468 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:42:40.018359 14468 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:42:40.020049 14383 master_runner.cc:428] Detected that this master e697954f7ed54a67a0a9a51adfc5e1ba is joining an existing cluster
I20260929 18:42:40.020112 14383 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.157.125:42879
I20260929 18:42:40.029407 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.022994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.7.157.125:42879)
0929 18:42:40.023106 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.023110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.023121 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:42:40.023147 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.023150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.023182 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.023253 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.023664 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.024145 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.024882 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.025126 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.025216 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.026382 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.026385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.026388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.026411 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.027964 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.028362 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.028365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.028367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.028441 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.028643 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.028647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.028649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.028809 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.028890 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.029114 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.029286 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":35}
I20260929 18:42:40.029414 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.022710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42879 (local address 127.0.0.1:37412)
0929 18:42:40.022952 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.022965 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.023042 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.023276 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.023279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.023469 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.023567 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.023573 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.024232 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.024235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.024787 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.024794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.025225 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.025228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.025344 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.025804 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.025818 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.026272 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.028052 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.028055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.028057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.028267 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.028511 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.028513 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.028514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.028569 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.028887 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.028890 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.028971 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.029052 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.029194 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":17,"thread_start_us":90,"threads_started":1}
I20260929 18:42:40.030994 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.022408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51912 (local address 127.7.157.126:45147)
0929 18:42:40.022525 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.022530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.022661 (+ 131us) server_negotiation.cc:407] Connection header received
0929 18:42:40.022797 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.022801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.022852 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.022931 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.023622 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.024434 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.025646 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.025967 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.026061 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.027343 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.027347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.027350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.027378 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.029026 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.029681 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.029685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.029687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.029775 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.030076 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.030080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.030082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.030294 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.030365 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.030721 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.030839 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:42:40.031446 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.022325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:51912)
0929 18:42:40.022585 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.022598 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.022695 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.023052 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.023056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.023356 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.023483 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.023492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.024570 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.024573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.025529 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.025540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.026069 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.026072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.026159 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.026619 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.026634 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.027183 (+ 549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.029147 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.029150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.029153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.029403 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.029863 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.029866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.029868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.029965 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.030432 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.030435 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.030530 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.031156 (+ 626us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.031290 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":90,"threads_started":1}
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707358, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.126@KRBTEST.COM for oryx/127.7.157.125@KRBTEST.COM
I20260929 18:42:40.043574 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.035540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.7.157.125:42879)
0929 18:42:40.036235 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.036238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.036248 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:42:40.036297 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.036299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.036339 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.036402 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.037027 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.037521 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.038357 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.038557 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.038637 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.040721 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.040724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.040726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.040747 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.041936 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.042377 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.042381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.042382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.042452 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.042682 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.042685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.042687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.042854 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.042936 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.043298 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.043417 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626}
I20260929 18:42:40.043576 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.035460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42879 (local address 127.0.0.1:37428)
0929 18:42:40.035841 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.035891 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.036009 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.036458 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.036464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.036814 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.036919 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.036925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.037616 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.037620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.038250 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.038259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.038647 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.038650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.038779 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.039259 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.039271 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.040594 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.042032 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.042041 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.042047 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.042290 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.042528 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.042531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.042533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.042609 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.042948 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.042955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.043094 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.043239 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.043425 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":111,"threads_started":1}
I20260929 18:42:40.045305 13919 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.157.125:42879: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba"
member_type: NON_VOTER
last_known_addr {
host: "127.7.157.125"
port: 42879
}
attrs {
promote: true
}
}
dest_uuid: "d3bffeef2ce1474981574d052413f6b6"
cas_config_opid_index: -1
I20260929 18:42:40.045976 13919 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } }
I20260929 18:42:40.048585 14425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:40.049752 14425 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 FOLLOWER]: Refusing update from remote peer d3bffeef2ce1474981574d052413f6b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 18:42:40.050192 14478 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e697954f7ed54a67a0a9a51adfc5e1ba (127.7.157.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } } }
I20260929 18:42:40.051225 14479 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:40.051812 14478 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.157.125:42879
I20260929 18:42:40.052266 14383 master.cc:537] Master@127.7.157.125:42879 shutting down...
I20260929 18:42:40.052757 14480 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } } }
I20260929 18:42:40.052955 14480 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: This master's current role is: LEADER
W20260929 18:42:40.053290 13905 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 18:42:40.054075 13905 consensus_peers.cc:487] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 -> Peer e697954f7ed54a67a0a9a51adfc5e1ba (127.7.157.125:42879): Couldn't send request to peer e697954f7ed54a67a0a9a51adfc5e1ba. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:42:40.056241 14383 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:40.056531 14383 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:40.056646 14383 tablet_replica.cc:330] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: stopping tablet replica
WARNING: no policy specified for oryx/127.7.157.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.157.124@KRBTEST.COM" created.
I20260929 18:42:40.071969 14383 master.cc:552] Master@127.7.157.125:42879 shutdown complete.
I20260929 18:42:40.072031 14383 master_runner.cc:305] Clearing existing system tablet
I20260929 18:42:40.072291 14383 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:40.072347 14383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:40.073721 14383 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:40.075368 14485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:40.075603 14383 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:40.075734 14383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:42:40.075773 14383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/wal
uuid: "e697954f7ed54a67a0a9a51adfc5e1ba"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "2413ed0cc59f0d3840ffc20e3c703c83"
server_key_iv: "c36c91c7a5beaee353ed452ea58877c6"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.157.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.124.keytab.
Entry for principal oryx/127.7.157.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.124.keytab.
I20260929 18:42:40.077823 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.124:34937
--webserver_interface=127.7.157.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.124.keytab
--principal=oryx/127.7.157.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:45147,127.7.157.125:42879,127.7.157.124:34937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:42:40.077782 14383 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:40.080077 14383 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 18:42:40.080166 14383 log.cc:1196] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 18:42:40.080402 14383 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: Deleting consensus metadata
I20260929 18:42:40.080570 14383 master_runner.cc:315] Copying system tablet from 127.7.157.126:45147
I20260929 18:42:40.081652 14383 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: tablet copy: Beginning tablet copy session from remote peer at address 127.7.157.126:45147
I20260929 18:42:40.089427 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.082321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51920 (local address 127.7.157.126:45147)
0929 18:42:40.082459 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.082464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.082702 (+ 238us) server_negotiation.cc:407] Connection header received
0929 18:42:40.082828 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.082831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.082882 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.082958 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.083471 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.084001 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.084825 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.085064 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.085163 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.086315 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.086319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.086322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.086349 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.087824 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.088242 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.088246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.088248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.088335 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.088596 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.088599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.088601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.088789 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.088881 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.089152 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.089259 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:42:40.089438 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.082194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:51920)
0929 18:42:40.082595 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.082610 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.082736 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.082991 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.082994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.083233 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.083346 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.083353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.084111 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.084115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.084705 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.084713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.085174 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.085178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.085271 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.085766 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.085782 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.086175 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.087932 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.087936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.087938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.088148 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.088419 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.088423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.088425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.088511 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.088879 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.088882 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.088990 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.089087 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.089233 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":102,"threads_started":1}
I20260929 18:42:40.090252 13939 tablet_copy_service.cc:140] P d3bffeef2ce1474981574d052413f6b6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e697954f7ed54a67a0a9a51adfc5e1ba ({username='oryx', principal='oryx/127.7.157.125@KRBTEST.COM'} at 127.0.0.1:51920)
I20260929 18:42:40.090420 13939 tablet_copy_service.cc:161] P d3bffeef2ce1474981574d052413f6b6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e697954f7ed54a67a0a9a51adfc5e1ba at {username='oryx', principal='oryx/127.7.157.125@KRBTEST.COM'} at 127.0.0.1:51920: session id = e697954f7ed54a67a0a9a51adfc5e1ba-00000000000000000000000000000000
I20260929 18:42:40.091953 13939 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 18:42:40.093427 14383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 18:42:40.095482 14383 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: tablet copy: Starting download of 0 data blocks...
I20260929 18:42:40.095667 14383 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: tablet copy: Starting download of 1 WAL segments...
I20260929 18:42:40.097313 14383 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 18:42:40.098906 13939 tablet_copy_service.cc:342] P d3bffeef2ce1474981574d052413f6b6: Request end of tablet copy session e697954f7ed54a67a0a9a51adfc5e1ba-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.157.125@KRBTEST.COM'} at 127.0.0.1:51920
I20260929 18:42:40.099030 13939 tablet_copy_service.cc:433] P d3bffeef2ce1474981574d052413f6b6: ending tablet copy session e697954f7ed54a67a0a9a51adfc5e1ba-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e697954f7ed54a67a0a9a51adfc5e1ba
I20260929 18:42:40.106278 14383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:40.107883 14493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:40.108135 14496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:40.108449 14383 server_base.cc:1034] running on GCE node
W20260929 18:42:40.111480 14494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:40.111709 14383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:40.111999 14383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:40.113142 14383 hybrid_clock.cc:648] HybridClock initialized: now 1790707360113130 us; error 29 us; skew 500 ppm
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:40.114745 14383 init.cc:377] Logged in from keytab as oryx/127.7.157.125@KRBTEST.COM (short username oryx)
I20260929 18:42:40.115289 14383 webserver.cc:460] Webserver started at http://127.7.157.125:33923/ using document root <none> and password file <none>
I20260929 18:42:40.115476 14383 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:40.115525 14383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:40.116726 14383 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:40.117658 14503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:40.117885 14383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:40.117977 14383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/wal
uuid: "e697954f7ed54a67a0a9a51adfc5e1ba"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "2413ed0cc59f0d3840ffc20e3c703c83"
server_key_iv: "c36c91c7a5beaee353ed452ea58877c6"
server_key_version: "encryptionkey@0"
I20260929 18:42:40.118065 14383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:40.123975 14383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:40.125864 14383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:40.130677 14383 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.125:42879
I20260929 18:42:40.130936 14555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.125:42879 every 8 connection(s)
I20260929 18:42:40.131273 14383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-1/data/info.pb
I20260929 18:42:40.132712 14556 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:42:40.133498 14556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: Bootstrap starting.
I20260929 18:42:40.141522 14556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:42:40.141803 14556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: Bootstrap complete.
I20260929 18:42:40.142302 14556 raft_consensus.cc:348] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } }
I20260929 18:42:40.142416 14556 raft_consensus.cc:729] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e697954f7ed54a67a0a9a51adfc5e1ba, State: Initialized, Role: LEARNER
I20260929 18:42:40.142514 14556 consensus_queue.cc:260] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } }
I20260929 18:42:40.142808 14559 sys_catalog.cc:455] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } } }
I20260929 18:42:40.142951 14556 sys_catalog.cc:564] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:40.142974 14559 sys_catalog.cc:458] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: This master's current role is: LEARNER
I20260929 18:42:40.145741 14570 catalog_manager.cc:1259] Loaded cluster ID: 50298e1848a94311b5e0994750e859f9
I20260929 18:42:40.145869 14570 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: loading cluster ID for follower catalog manager: success
I20260929 18:42:40.147470 14570 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: acquiring CA information for follower catalog manager: success
I20260929 18:42:40.148557 14570 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 18:42:40.229727 14486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:40.230016 14486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:40.230060 14486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:40.234026 14486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:40.234102 14486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:40.234123 14486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:40.234153 14486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:40.234170 14486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:40.239570 14486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:45147,127.7.157.125:42879,127.7.157.124:34937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.124.keytab
--principal=oryx/127.7.157.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.124:34937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.157.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:40.240803 14486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:40.241932 14486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:40.250495 14575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:40.250483 14578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:40.250483 14576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:40.250909 14486 server_base.cc:1034] running on GCE node
I20260929 18:42:40.251492 14486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:40.252642 14486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:40.253794 14486 hybrid_clock.cc:648] HybridClock initialized: now 1790707360253767 us; error 45 us; skew 500 ppm
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:40.256395 14486 init.cc:377] Logged in from keytab as oryx/127.7.157.124@KRBTEST.COM (short username oryx)
I20260929 18:42:40.257275 14486 webserver.cc:460] Webserver started at http://127.7.157.124:39425/ using document root <none> and password file <none>
I20260929 18:42:40.257815 14486 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:40.257879 14486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:40.258068 14486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:40.260181 14486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/data/instance:
uuid: "fa02076e43f842069ef4502c6855b606"
format_stamp: "Formatted at 2026-09-29 18:42:40 on dist-test-slave-smpn"
server_key: "e6c96853e31c75d8788ffd958a1974da"
server_key_iv: "923815d82135bb1241a77c678d352034"
server_key_version: "encryptionkey@0"
I20260929 18:42:40.260721 14486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/wal/instance:
uuid: "fa02076e43f842069ef4502c6855b606"
format_stamp: "Formatted at 2026-09-29 18:42:40 on dist-test-slave-smpn"
server_key: "e6c96853e31c75d8788ffd958a1974da"
server_key_iv: "923815d82135bb1241a77c678d352034"
server_key_version: "encryptionkey@0"
I20260929 18:42:40.264354 14486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:42:40.266641 14585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:40.267918 14486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:42:40.268025 14486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/wal
uuid: "fa02076e43f842069ef4502c6855b606"
format_stamp: "Formatted at 2026-09-29 18:42:40 on dist-test-slave-smpn"
server_key: "e6c96853e31c75d8788ffd958a1974da"
server_key_iv: "923815d82135bb1241a77c678d352034"
server_key_version: "encryptionkey@0"
I20260929 18:42:40.268118 14486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:40.283980 14486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:40.286898 14486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:40.287106 14486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:40.293970 14486 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.124:34937
I20260929 18:42:40.294243 14637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.124:34937 every 8 connection(s)
I20260929 18:42:40.295219 14486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/data/info.pb
I20260929 18:42:40.298839 14638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:40.303164 14638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:42:40.304992 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 14486
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.124@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.157.124@KRBTEST.COM
I20260929 18:42:40.316833 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.305127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.7.157.126:45147)
0929 18:42:40.305281 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.305286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.306035 (+ 749us) server_negotiation.cc:407] Connection header received
0929 18:42:40.307025 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.307029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.307102 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.307184 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.308196 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.309062 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.310389 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.310736 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.310837 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.313450 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.313454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.313456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.313482 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.314858 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.315382 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.315385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.315386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.315457 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.315770 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.315773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.315775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.315950 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.316035 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.316542 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.316659 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:42:40.317770 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.305731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.7.157.124:34937)
0929 18:42:40.306084 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.306097 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.306229 (+ 132us) server_negotiation.cc:407] Connection header received
0929 18:42:40.306860 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.306882 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.307157 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.307328 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.308128 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.309344 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.310413 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.310947 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.311021 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.313830 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.313835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.313846 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.313865 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.315445 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.315832 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.315838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.315839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.315908 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.316149 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.316152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.316154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.316386 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.316554 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.316757 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.317002 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":133,"threads_started":1}
I20260929 18:42:40.317775 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.305421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:51922)
0929 18:42:40.305895 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.305946 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.306844 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.307236 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.307243 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.307700 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.308052 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.308071 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.309173 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.309175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.310265 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.310274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.310866 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.310868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.310995 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.311593 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.311615 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.313295 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.314954 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.314961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.314971 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.315296 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.315541 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.315544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.315547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.315682 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.316045 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.316052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.316328 (+ 276us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.316488 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.316798 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":125,"threads_started":1}
I20260929 18:42:40.319371 14638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.124@KRBTEST.COM for oryx/127.7.157.125@KRBTEST.COM
I20260929 18:42:40.326524 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.319818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42879 (local address 127.0.0.1:37442)
0929 18:42:40.319966 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.319980 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.320049 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.320321 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.320323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.320441 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.320534 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.320539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.321370 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.321372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.321903 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.321909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.322305 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.322307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.322373 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.322788 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.322801 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.323966 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.325385 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.325387 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.325389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.325564 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.325775 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.325776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.325778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.325840 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.326127 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.326129 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.326205 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.326277 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.326377 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 18:42:40.326591 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.319898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37442 (local address 127.7.157.125:42879)
0929 18:42:40.320146 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.320149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.320160 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:42:40.320196 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.320198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.320234 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.320304 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.320622 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.321292 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.321987 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.322216 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.322301 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.324103 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.324106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.324109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.324133 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.325290 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.325635 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.325637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.325638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.325710 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.325889 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.325891 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.325893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.326059 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.326134 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.326309 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.326439 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
I20260929 18:42:40.327252 14638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.124@KRBTEST.COM for oryx/127.7.157.124@KRBTEST.COM
I20260929 18:42:40.334355 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.327710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:34937 (local address 127.0.0.1:37672)
0929 18:42:40.327832 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.327846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.327911 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.328089 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.328091 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.328210 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.328279 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.328283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.328908 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.328910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.329397 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.329403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.329779 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.329781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.329834 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.330173 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.330187 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.331464 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.332933 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.332936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.332938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.333167 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.333386 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.333390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.333392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.333475 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.333819 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.333822 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.333913 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.334029 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.334169 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 18:42:40.334357 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.327770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37672 (local address 127.7.157.124:34937)
0929 18:42:40.327869 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.327872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.327883 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:42:40.327972 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.327974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.328007 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.328064 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.328351 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.328830 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.329469 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.329693 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.329781 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.331587 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.331590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.331592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.331612 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.332835 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.333248 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.333250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.333252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.333315 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.333559 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.333561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.333562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.333716 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.333786 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.334071 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.334228 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":39}
I20260929 18:42:40.337620 14638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: Bootstrap starting.
I20260929 18:42:40.339912 14638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:40.340580 14638 log.cc:824] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:40.342387 14638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: No bootstrap required, opened a new log
I20260929 18:42:40.345384 14638 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } }
I20260929 18:42:40.345585 14638 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:40.345618 14638 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa02076e43f842069ef4502c6855b606, State: Initialized, Role: FOLLOWER
I20260929 18:42:40.345999 14638 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } }
I20260929 18:42:40.346427 14644 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } } }
I20260929 18:42:40.346546 14644 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:42:40.347232 14638 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:42:40.351647 14655 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:42:40.351732 14655 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:42:40.352854 14110 heartbeater.cc:503] Master 127.7.157.126:45147 was elected leader, sending a full tablet report...
I20260929 18:42:40.354934 14486 master_runner.cc:428] Detected that this master fa02076e43f842069ef4502c6855b606 is joining an existing cluster
I20260929 18:42:40.354983 14486 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.157.124:34937
I20260929 18:42:40.365589 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.356940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.7.157.126:45147)
0929 18:42:40.357051 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.357054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.357907 (+ 853us) server_negotiation.cc:407] Connection header received
0929 18:42:40.357990 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.357993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.358037 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.358099 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.358501 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.359061 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.360117 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.360448 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.360545 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.362057 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.362061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.362064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.362090 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.363740 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.364277 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.364280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.364282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.364364 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.364602 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.364605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.364607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.364816 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.364901 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.365351 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.365452 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:42:40.365602 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.357082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:51936)
0929 18:42:40.357827 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.357838 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.357920 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.358121 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.358123 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.358306 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.358400 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.358405 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.359225 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.359229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.360017 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.360026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.360732 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.360735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.360935 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.361475 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.361489 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.361930 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.363839 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.363842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.363845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.364164 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.364444 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.364447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.364449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.364526 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.364890 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.364893 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.364991 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.365280 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.365417 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":72,"spinlock_wait_cycles":768,"thread_start_us":108,"threads_started":1}
I20260929 18:42:40.367383 14662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.357598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42879 (local address 127.0.0.1:37450)
0929 18:42:40.360374 (+ 2776us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.360386 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.360505 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.360833 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.360836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.361059 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.361167 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.361173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.361865 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.361868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.362634 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.362646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.363258 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.363261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.363390 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.363829 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.363844 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.364237 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.365928 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.365931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.365933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.366149 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.366426 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.366429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.366431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.366522 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.366899 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.366902 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.366984 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.367095 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.367217 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2717,"thread_start_us":70,"threads_started":1}
I20260929 18:42:40.367400 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.357484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.7.157.125:42879)
0929 18:42:40.357607 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.357611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.360448 (+ 2837us) server_negotiation.cc:407] Connection header received
0929 18:42:40.360604 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.360607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.360653 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.360700 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.361275 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.361763 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.362753 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.363138 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.363248 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.364342 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.364346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.364349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.364375 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.365835 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.366265 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.366268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.366270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.366350 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.366610 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.366613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.366614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.366818 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.366907 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.367147 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.367269 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:42:40.370935 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 13981
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707358, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.126@KRBTEST.COM for oryx/127.7.157.124@KRBTEST.COM
I20260929 18:42:40.377950 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:45901
--local_ip_for_outbound_sockets=127.7.157.65
--tserver_master_addrs=127.7.157.126:45147
--webserver_port=38045
--webserver_interface=127.7.157.65
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.65.keytab
--principal=oryx/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.157.126:45147,127.7.157.125:42879
--tserver_master_addrs=127.7.157.126:45147,127.7.157.125:42879,127.7.157.124:34937 with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:42:40.386473 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.369393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:34937 (local address 127.0.0.1:37700)
0929 18:42:40.369550 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.369563 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.369634 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.369935 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.369938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.370101 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.370191 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.370197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.371150 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.371154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.371954 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.371964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.372542 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.372545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.372635 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.373077 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.373090 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.374560 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.384922 (+ 10362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.384926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.384928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.385173 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.385474 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.385478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.385480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.385563 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.385967 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.385971 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.386066 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.386141 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.386302 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 18:42:40.387018 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.369450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37700 (local address 127.7.157.124:34937)
0929 18:42:40.369706 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.369716 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.369729 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:42:40.369787 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.369790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.369836 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.369921 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.370304 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.370950 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.372089 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.372426 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.373643 (+ 1217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.374718 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.374722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.374725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.374747 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.384785 (+ 10038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.385298 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.385301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.385303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.385381 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.385658 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.385661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.385663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.385866 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.386654 (+ 788us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.386727 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.386845 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":119,"threads_started":1}
I20260929 18:42:40.388109 13919 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.157.124:34937: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fa02076e43f842069ef4502c6855b606"
member_type: NON_VOTER
last_known_addr {
host: "127.7.157.124"
port: 34937
}
attrs {
promote: true
}
}
dest_uuid: "d3bffeef2ce1474981574d052413f6b6"
cas_config_opid_index: 5
I20260929 18:42:40.388738 13919 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } }
I20260929 18:42:40.388978 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.356998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:34937 (local address 127.0.0.1:37686)
0929 18:42:40.357265 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.357279 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.357379 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.357704 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.357707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.357933 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.358056 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.358063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.359133 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.359136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.360143 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.360153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.360732 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.360735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.360923 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.384499 (+ 23576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.384517 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.385064 (+ 547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.387542 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.387545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.387547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.387792 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.388067 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.388070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.388072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.388141 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.388526 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.388529 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.388600 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.388673 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.388800 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":70,"spinlock_wait_cycles":2304,"thread_start_us":92,"threads_started":1}
I20260929 18:42:40.389469 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.357337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37686 (local address 127.7.157.124:34937)
0929 18:42:40.357433 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.357438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.357451 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:42:40.357485 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.357488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.357531 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.357605 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.358170 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.358962 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.360258 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.360543 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.360620 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.385226 (+ 24606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.385229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.385233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.385257 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.386786 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.387899 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.387902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.387904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.387981 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.388228 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.388231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.388233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.388412 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.388495 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.389244 (+ 749us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.389344 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:42:40.389819 14479 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fa02076e43f842069ef4502c6855b606 (127.7.157.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } } }
I20260929 18:42:40.391489 14612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:40.392805 14612 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 FOLLOWER]: Refusing update from remote peer d3bffeef2ce1474981574d052413f6b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 18:42:40.393357 14478 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:40.394994 14479 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.157.124:34937
I20260929 18:42:40.395013 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.389895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42879 (local address 127.0.0.1:37456)
0929 18:42:40.390538 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.390551 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.390631 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.390974 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.390977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.390994 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:40.391102 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.391108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.392205 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.392209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.393489 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.393499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.394604 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.394607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.394698 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.394749 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.394819 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.394890 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568}
I20260929 18:42:40.395254 14479 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } } }
I20260929 18:42:40.395354 14479 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:40.395380 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.390117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.7.157.125:42879)
0929 18:42:40.390230 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.390234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.390617 (+ 383us) server_negotiation.cc:407] Connection header received
0929 18:42:40.390708 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.390711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.390770 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.390857 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:40.391304 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.392080 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.393625 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.394289 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.394385 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.394548 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.395177 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.395269 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:42:40.395511 14486 master.cc:537] Master@127.7.157.124:34937 shutting down...
I20260929 18:42:40.396163 14522 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 LEARNER]: Refusing update from remote peer d3bffeef2ce1474981574d052413f6b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 18:42:40.396633 14480 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 18:42:40.398488 14522 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fa02076e43f842069ef4502c6855b606 (127.7.157.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } } }
I20260929 18:42:40.402221 14612 raft_consensus.cc:858] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } }
I20260929 18:42:40.402829 14559 sys_catalog.cc:455] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3bffeef2ce1474981574d052413f6b6. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } } }
I20260929 18:42:40.402948 14559 sys_catalog.cc:458] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: This master's current role is: LEARNER
I20260929 18:42:40.403034 14559 sys_catalog.cc:455] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } } }
I20260929 18:42:40.403160 14559 sys_catalog.cc:458] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: This master's current role is: LEARNER
W20260929 18:42:40.403362 14612 raft_consensus.cc:1565] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } }
I20260929 18:42:40.405138 14612 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e697954f7ed54a67a0a9a51adfc5e1ba (127.7.157.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } } }
I20260929 18:42:40.407482 14670 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } } }
I20260929 18:42:40.407583 14670 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 18:42:40.407660 14670 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } } }
I20260929 18:42:40.407725 14670 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 18:42:40.407857 14644 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3bffeef2ce1474981574d052413f6b6. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } } }
I20260929 18:42:40.407924 14644 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 18:42:40.409451 14480 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6: attempting to promote NON_VOTER e697954f7ed54a67a0a9a51adfc5e1ba to VOTER
I20260929 18:42:40.409756 14486 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260929 18:42:40.409986 14486 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260929 18:42:40.410063 14486 tablet_replica.cc:330] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: stopping tablet replica
I20260929 18:42:40.410616 14480 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } }
I20260929 18:42:40.412305 14522 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 LEARNER]: Refusing update from remote peer d3bffeef2ce1474981574d052413f6b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260929 18:42:40.412582 13903 consensus_peers.cc:487] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 -> Peer fa02076e43f842069ef4502c6855b606 (127.7.157.124:34937): Couldn't send request to peer fa02076e43f842069ef4502c6855b606. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:42:40.412706 14479 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 18:42:40.412860 14478 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: NON_VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: true } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } } }
I20260929 18:42:40.412990 14478 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:40.414319 14479 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e697954f7ed54a67a0a9a51adfc5e1ba (127.7.157.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } } }
I20260929 18:42:40.414759 14522 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e697954f7ed54a67a0a9a51adfc5e1ba (127.7.157.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } } }
I20260929 18:42:40.415663 14479 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } } }
I20260929 18:42:40.415751 14479 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:40.415849 14479 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } } }
I20260929 18:42:40.415908 14479 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:40.416213 14667 sys_catalog.cc:455] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } } }
I20260929 18:42:40.416309 14667 sys_catalog.cc:458] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:42:40.428987 14486 master.cc:552] Master@127.7.157.124:34937 shutdown complete.
I20260929 18:42:40.429052 14486 master_runner.cc:305] Clearing existing system tablet
I20260929 18:42:40.429298 14486 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:40.429355 14486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:40.430678 14486 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:42:40.431617 14677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:40.431890 14486 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:40.432009 14486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:42:40.432039 14486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/wal
uuid: "fa02076e43f842069ef4502c6855b606"
format_stamp: "Formatted at 2026-09-29 18:42:40 on dist-test-slave-smpn"
server_key: "e6c96853e31c75d8788ffd958a1974da"
server_key_iv: "923815d82135bb1241a77c678d352034"
server_key_version: "encryptionkey@0"
I20260929 18:42:40.433892 14486 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:40.436318 14486 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 18:42:40.436406 14486 log.cc:1196] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 18:42:40.436695 14486 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: Deleting consensus metadata
I20260929 18:42:40.436848 14486 master_runner.cc:315] Copying system tablet from 127.7.157.126:45147
I20260929 18:42:40.437958 14486 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: tablet copy: Beginning tablet copy session from remote peer at address 127.7.157.126:45147
I20260929 18:42:40.446698 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.438597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51938 (local address 127.7.157.126:45147)
0929 18:42:40.438720 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.438726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.438888 (+ 162us) server_negotiation.cc:407] Connection header received
0929 18:42:40.439007 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.439010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.439061 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.439171 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.439655 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.440464 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.441579 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.441919 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.442022 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.443377 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.443381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.443384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.443412 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.445098 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.445532 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.445535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.445537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.445620 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.445879 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.445882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.445883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.446086 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.446178 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.446424 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.446537 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:42:40.447422 14683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.438491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:51938)
0929 18:42:40.438795 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.438809 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.438924 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.439213 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.439216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.439425 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.439541 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.439548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.440580 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.440584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.441460 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.441471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.442043 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.442046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.442130 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.442677 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.442691 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.443181 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.445195 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.445199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.445201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.445443 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.445701 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.445712 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.445714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.445801 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.446171 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.446174 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.446271 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.446370 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.447262 (+ 892us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":87,"threads_started":1}
I20260929 18:42:40.448103 13939 tablet_copy_service.cc:140] P d3bffeef2ce1474981574d052413f6b6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fa02076e43f842069ef4502c6855b606 ({username='oryx', principal='oryx/127.7.157.124@KRBTEST.COM'} at 127.0.0.1:51938)
I20260929 18:42:40.448203 13939 tablet_copy_service.cc:161] P d3bffeef2ce1474981574d052413f6b6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fa02076e43f842069ef4502c6855b606 at {username='oryx', principal='oryx/127.7.157.124@KRBTEST.COM'} at 127.0.0.1:51938: session id = fa02076e43f842069ef4502c6855b606-00000000000000000000000000000000
I20260929 18:42:40.449303 13939 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 18:42:40.450582 14486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 18:42:40.453109 14486 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: tablet copy: Starting download of 0 data blocks...
I20260929 18:42:40.453375 14486 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: tablet copy: Starting download of 1 WAL segments...
I20260929 18:42:40.455010 14486 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 18:42:40.456876 13939 tablet_copy_service.cc:342] P d3bffeef2ce1474981574d052413f6b6: Request end of tablet copy session fa02076e43f842069ef4502c6855b606-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.157.124@KRBTEST.COM'} at 127.0.0.1:51938
I20260929 18:42:40.456961 13939 tablet_copy_service.cc:433] P d3bffeef2ce1474981574d052413f6b6: ending tablet copy session fa02076e43f842069ef4502c6855b606-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fa02076e43f842069ef4502c6855b606
I20260929 18:42:40.465008 14486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:40.466785 14684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:40.467865 14685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:40.468236 14687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:40.472024 14486 server_base.cc:1034] running on GCE node
I20260929 18:42:40.472239 14486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:40.472548 14486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:40.473696 14486 hybrid_clock.cc:648] HybridClock initialized: now 1790707360473676 us; error 39 us; skew 500 ppm
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:40.475387 14486 init.cc:377] Logged in from keytab as oryx/127.7.157.124@KRBTEST.COM (short username oryx)
I20260929 18:42:40.475881 14486 webserver.cc:460] Webserver started at http://127.7.157.124:44111/ using document root <none> and password file <none>
I20260929 18:42:40.476074 14486 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:40.476117 14486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:40.477653 14486 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:40.478685 14694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:40.478968 14486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:40.479048 14486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/wal
uuid: "fa02076e43f842069ef4502c6855b606"
format_stamp: "Formatted at 2026-09-29 18:42:40 on dist-test-slave-smpn"
server_key: "e6c96853e31c75d8788ffd958a1974da"
server_key_iv: "923815d82135bb1241a77c678d352034"
server_key_version: "encryptionkey@0"
I20260929 18:42:40.479171 14486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:40.511415 14486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
W20260929 18:42:40.511902 14665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:40.512183 14665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:40.512259 14665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 18:42:40.513377 14486 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260929 18:42:40.517302 14665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:40.517357 14665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:40.517441 14665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:42:40.518405 14486 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.124:34937
I20260929 18:42:40.519044 14486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/master-2/data/info.pb
I20260929 18:42:40.519482 14749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.124:34937 every 8 connection(s)
I20260929 18:42:40.520829 14750 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:42:40.521898 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: Bootstrap starting.
I20260929 18:42:40.524678 14665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.65.keytab
--principal=oryx/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:45901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=38045
--tserver_master_addrs=127.7.157.126:45147,127.7.157.125:42879,127.7.157.124:34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:40.525908 14665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:40.527237 14665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:42:40.529779 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:42:40.530035 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: Bootstrap complete.
I20260929 18:42:40.530530 14750 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } }
I20260929 18:42:40.530663 14750 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa02076e43f842069ef4502c6855b606, State: Initialized, Role: LEARNER
I20260929 18:42:40.530781 14750 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } }
I20260929 18:42:40.531096 14753 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } } }
I20260929 18:42:40.531207 14753 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: This master's current role is: LEARNER
I20260929 18:42:40.531234 14750 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:40.534425 14764 catalog_manager.cc:1259] Loaded cluster ID: 50298e1848a94311b5e0994750e859f9
I20260929 18:42:40.534479 14764 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: loading cluster ID for follower catalog manager: success
I20260929 18:42:40.536440 14764 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: acquiring CA information for follower catalog manager: success
I20260929 18:42:40.537710 14764 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 18:42:40.539016 14769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:40.539131 14665 server_base.cc:1034] running on GCE node
W20260929 18:42:40.539049 14766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:40.539038 14767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:40.539623 14665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:40.540235 14665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:40.541385 14665 hybrid_clock.cc:648] HybridClock initialized: now 1790707360541370 us; error 28 us; skew 500 ppm
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:40.544047 14665 init.cc:377] Logged in from keytab as oryx/127.7.157.65@KRBTEST.COM (short username oryx)
I20260929 18:42:40.544862 14665 webserver.cc:460] Webserver started at http://127.7.157.65:38045/ using document root <none> and password file <none>
I20260929 18:42:40.545378 14665 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:40.545449 14665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:40.549484 14665 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 18:42:40.551903 14776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:40.553045 14665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:42:40.553133 14665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "5a8d904fb2c54e72bbf064ec58d2a15a"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "f3005f7f3333f18bede5e37a25975fdd"
server_key_iv: "89a4195fa48c8603a70d937cb7faac33"
server_key_version: "encryptionkey@0"
I20260929 18:42:40.553627 14665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:40.569824 14244 heartbeater.cc:503] Master 127.7.157.126:45147 was elected leader, sending a full tablet report...
I20260929 18:42:40.575580 14665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:40.578193 14665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:40.578362 14665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:40.578925 14665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:40.579787 14665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:40.579833 14665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:40.579874 14665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:40.579902 14665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:40.588609 14665 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:45901
I20260929 18:42:40.588631 14889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:45901 every 8 connection(s)
I20260929 18:42:40.589572 14665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:42:40.594738 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 14665
I20260929 18:42:40.594934 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 14115
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.125@KRBTEST.COM
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.124@KRBTEST.COM
I20260929 18:42:40.601708 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:42467
--local_ip_for_outbound_sockets=127.7.157.66
--tserver_master_addrs=127.7.157.126:45147
--webserver_port=34275
--webserver_interface=127.7.157.66
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.66.keytab
--principal=oryx/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.157.126:45147,127.7.157.125:42879
--tserver_master_addrs=127.7.157.126:45147,127.7.157.125:42879,127.7.157.124:34937 with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:42:40.606174 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.592837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:33249 (local address 127.7.157.125:42879)
0929 18:42:40.592949 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.592953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.592989 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:42:40.593364 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.593367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.593415 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.593465 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.594881 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.595717 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.597607 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.597946 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.598016 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.602030 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.602033 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.602036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.602065 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.603538 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.604421 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.604424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.604426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.604507 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.604877 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.604880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.604882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.605087 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.605153 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.605901 (+ 748us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.606016 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:42:40.606174 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.591245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:46069 (local address 127.7.157.126:45147)
0929 18:42:40.591387 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.591392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.592331 (+ 939us) server_negotiation.cc:407] Connection header received
0929 18:42:40.593360 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.593363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.593415 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.593465 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.594882 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.595710 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.597608 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.597946 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.598016 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.601517 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.601521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.601525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.601554 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.603555 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.604587 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.604591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.604593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.604680 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.604955 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.604958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.604960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.605140 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.605200 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.605901 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.606016 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:42:40.607914 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.592689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42879 (local address 127.7.157.65:33249)
0929 18:42:40.592923 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.592935 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.593147 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.593660 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.593667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.594084 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.594725 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.594743 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.595956 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.595959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.597477 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.597488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.598205 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.598209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.598472 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.599149 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.599172 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.601889 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.603664 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.603669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.603686 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.604310 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.604608 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.604610 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.604612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.604783 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.605233 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.605240 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.605619 (+ 379us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.606405 (+ 786us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.606792 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":365,"spinlock_wait_cycles":33920,"thread_start_us":80,"threads_started":1}
I20260929 18:42:40.608160 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.591719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:34937 (local address 127.7.157.65:36985)
0929 18:42:40.592178 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.592229 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.593143 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.593659 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.593666 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.594078 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.594728 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.594743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.596305 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.596308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.597466 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.597477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.598205 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.598209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.598433 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.599586 (+ 1153us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.599600 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.601609 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.605111 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.605114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.605116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.605375 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.605682 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.605685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.605687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.605775 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.606888 (+ 1113us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.606893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.606993 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.607135 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.607279 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":280,"spinlock_wait_cycles":32256,"thread_start_us":106,"threads_started":1}
I20260929 18:42:40.607914 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.591719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.65:46069)
0929 18:42:40.592178 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.592230 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.593141 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.593660 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.593667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.594080 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.594726 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.594743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.596380 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.596383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.597477 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.597488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.598216 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.598220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.598460 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.599306 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.599319 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.601306 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.603664 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.603670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.603686 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.604481 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.604784 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.604787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.604789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.604865 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.605272 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.605275 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.605619 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.606405 (+ 786us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.606853 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":469,"spinlock_wait_cycles":23424,"thread_start_us":106,"threads_started":1}
I20260929 18:42:40.608770 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.591249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:36985 (local address 127.7.157.124:34937)
0929 18:42:40.591485 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.591497 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.592329 (+ 832us) server_negotiation.cc:407] Connection header received
0929 18:42:40.593348 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.593352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.593409 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.593486 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.594946 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.595942 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.597595 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.597970 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.598061 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.602943 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.602947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.602949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.602978 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.604996 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.605486 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.605489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.605491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.605581 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.605872 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.605876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.605877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.606085 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.606173 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.608486 (+ 2313us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.608611 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260929 18:42:40.609314 14891 heartbeater.cc:347] Connected to a master server at 127.7.157.126:45147
I20260929 18:42:40.609411 14890 heartbeater.cc:347] Connected to a master server at 127.7.157.124:34937
I20260929 18:42:40.609549 14890 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:40.609570 14892 heartbeater.cc:347] Connected to a master server at 127.7.157.125:42879
I20260929 18:42:40.609690 14892 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:40.609549 14891 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:40.610055 14891 heartbeater.cc:511] Master 127.7.157.126:45147 requested a full tablet report, sending...
I20260929 18:42:40.610105 14890 heartbeater.cc:511] Master 127.7.157.124:34937 requested a full tablet report, sending...
I20260929 18:42:40.610060 14892 heartbeater.cc:511] Master 127.7.157.125:42879 requested a full tablet report, sending...
I20260929 18:42:40.611369 13919 ts_manager.cc:194] Re-registered known tserver with Master: 5a8d904fb2c54e72bbf064ec58d2a15a (127.7.157.65:45901)
I20260929 18:42:40.611557 14520 ts_manager.cc:194] Registered new tserver with Master: 5a8d904fb2c54e72bbf064ec58d2a15a (127.7.157.65:45901)
I20260929 18:42:40.611557 14705 ts_manager.cc:194] Registered new tserver with Master: 5a8d904fb2c54e72bbf064ec58d2a15a (127.7.157.65:45901)
I20260929 18:42:40.612114 13919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:46069
W20260929 18:42:40.713238 14898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:40.713485 14898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:40.713528 14898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:40.717434 14898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:40.717487 14898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:40.717553 14898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:42:40.722679 14898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.66.keytab
--principal=oryx/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:42467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=34275
--tserver_master_addrs=127.7.157.126:45147,127.7.157.125:42879,127.7.157.124:34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:40.723697 14898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:40.724653 14898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:40.733847 14906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:40.733814 14903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:40.733808 14904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:40.734071 14898 server_base.cc:1034] running on GCE node
I20260929 18:42:40.734428 14898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:40.735049 14898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:40.736232 14898 hybrid_clock.cc:648] HybridClock initialized: now 1790707360736215 us; error 28 us; skew 500 ppm
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:40.738780 14898 init.cc:377] Logged in from keytab as oryx/127.7.157.66@KRBTEST.COM (short username oryx)
I20260929 18:42:40.739629 14898 webserver.cc:460] Webserver started at http://127.7.157.66:34275/ using document root <none> and password file <none>
I20260929 18:42:40.740154 14898 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:40.740229 14898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:40.743942 14898 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260929 18:42:40.745883 14913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:40.747097 14898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:40.747205 14898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "4ec3bc489671458189f218079f60c6ff"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "a4f31bd3c32c5b13dbd0222b1fa34d0e"
server_key_iv: "7e36442bc5d64c8efb4b32051c3610a7"
server_key_version: "encryptionkey@0"
I20260929 18:42:40.747756 14898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:40.759424 14898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:40.762115 14898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:40.762291 14898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:40.762845 14898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:40.763705 14898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:40.763768 14898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:40.763828 14898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:40.763855 14898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:40.772060 14898 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:42467
I20260929 18:42:40.772084 15026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:42467 every 8 connection(s)
I20260929 18:42:40.772986 14898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:42:40.773401 14378 heartbeater.cc:503] Master 127.7.157.126:45147 was elected leader, sending a full tablet report...
I20260929 18:42:40.778375 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 14898
I20260929 18:42:40.778553 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 14249
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.124@KRBTEST.COM
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.125@KRBTEST.COM
I20260929 18:42:40.786581 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:37959
--local_ip_for_outbound_sockets=127.7.157.67
--tserver_master_addrs=127.7.157.126:45147
--webserver_port=46613
--webserver_interface=127.7.157.67
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.67.keytab
--principal=oryx/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.157.126:45147,127.7.157.125:42879
--tserver_master_addrs=127.7.157.126:45147,127.7.157.125:42879,127.7.157.124:34937 with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:42:40.789178 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.775194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:49405 (local address 127.7.157.124:34937)
0929 18:42:40.775333 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.775337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.775720 (+ 383us) server_negotiation.cc:407] Connection header received
0929 18:42:40.776759 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.776763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.776819 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.776894 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.778239 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.778983 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.780938 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.781292 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.781383 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.785046 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.785050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.785052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.785079 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.786673 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.787235 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.787238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.787240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.787315 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.787627 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.787630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.787632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.787816 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.787907 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.788894 (+ 987us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.789021 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:42:40.790009 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.775136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:34937 (local address 127.7.157.66:49405)
0929 18:42:40.775587 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.775637 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.776567 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.777056 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.777062 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.777505 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.778066 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.778079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.779169 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.779172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.780813 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.780823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.781515 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.781519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.781887 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.782815 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.782828 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.784866 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.786787 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.786793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.786808 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.787124 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.787398 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.787401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.787403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.787533 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.787963 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.787970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.788281 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.788465 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.788930 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":510,"spinlock_wait_cycles":23424,"thread_start_us":156,"threads_started":1}
I20260929 18:42:40.790505 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.774652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:43763 (local address 127.7.157.125:42879)
0929 18:42:40.774806 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.774811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.775729 (+ 918us) server_negotiation.cc:407] Connection header received
0929 18:42:40.776783 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.776786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.776832 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.776881 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.778219 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.779022 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.780928 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.781262 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.781332 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.785251 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.785256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.785259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.785289 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.787805 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.789104 (+ 1299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.789107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.789109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.789184 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.789609 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.789613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.789614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.789838 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.789907 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.790253 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.790371 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:42:40.790681 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.776180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:47481 (local address 127.7.157.126:45147)
0929 18:42:40.776298 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.776302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.776376 (+ 74us) server_negotiation.cc:407] Connection header received
0929 18:42:40.776779 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.776783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.776832 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.776881 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.778226 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.779019 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.780928 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.781270 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.781332 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.785794 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.785797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.785800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.785828 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.788353 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.788907 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.788910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.788912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.788991 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.789591 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.789594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.789596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.789802 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.789889 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.790425 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.790541 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:42:40.790988 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.775136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42879 (local address 127.7.157.66:43763)
0929 18:42:40.775587 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.775637 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.776566 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.777056 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.777064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.777507 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.778076 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.778085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.779223 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.779226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.780809 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.780815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.781515 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.781519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.781883 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.782924 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.782937 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.785087 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.788457 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.788460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.788461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.788798 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.789286 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.789289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.789291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.789367 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.789974 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.789976 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.790067 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.790672 (+ 605us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.790822 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":383,"spinlock_wait_cycles":47488,"thread_start_us":107,"threads_started":1}
I20260929 18:42:40.791312 15027 heartbeater.cc:347] Connected to a master server at 127.7.157.124:34937
I20260929 18:42:40.791538 15027 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:40.791574 15029 heartbeater.cc:347] Connected to a master server at 127.7.157.125:42879
I20260929 18:42:40.791632 15029 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:40.792133 15029 heartbeater.cc:511] Master 127.7.157.125:42879 requested a full tablet report, sending...
I20260929 18:42:40.792133 15027 heartbeater.cc:511] Master 127.7.157.124:34937 requested a full tablet report, sending...
I20260929 18:42:40.792325 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.776079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.66:47481)
0929 18:42:40.776312 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.776322 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.776578 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.777056 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.777064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.777507 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.778078 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.778084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.779171 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.779174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.780800 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.780810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.781515 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.781519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.781883 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.782653 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.782673 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.785623 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.788455 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.788458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.788460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.788834 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.789438 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.789441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.789442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.789504 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.789930 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.789933 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.790015 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.791291 (+ 1276us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.791469 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":454,"spinlock_wait_cycles":25088,"thread_start_us":73,"threads_started":1}
I20260929 18:42:40.793349 15028 heartbeater.cc:347] Connected to a master server at 127.7.157.126:45147
I20260929 18:42:40.793385 14705 ts_manager.cc:194] Registered new tserver with Master: 4ec3bc489671458189f218079f60c6ff (127.7.157.66:42467)
I20260929 18:42:40.793470 15028 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:40.793627 15028 heartbeater.cc:511] Master 127.7.157.126:45147 requested a full tablet report, sending...
I20260929 18:42:40.793668 14520 ts_manager.cc:194] Registered new tserver with Master: 4ec3bc489671458189f218079f60c6ff (127.7.157.66:42467)
I20260929 18:42:40.794170 13919 ts_manager.cc:194] Re-registered known tserver with Master: 4ec3bc489671458189f218079f60c6ff (127.7.157.66:42467)
I20260929 18:42:40.794694 13919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:47481
W20260929 18:42:40.897679 15034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:40.897953 15034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:40.898012 15034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:40.902066 15034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:40.902137 15034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:40.902218 15034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:42:40.907675 15034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.67.keytab
--principal=oryx/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:37959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=46613
--tserver_master_addrs=127.7.157.126:45147,127.7.157.125:42879,127.7.157.124:34937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:40.908881 15034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:40.909878 15034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:42:40.911718 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.908246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:34937 (local address 127.0.0.1:37712)
0929 18:42:40.908481 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.908495 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.908629 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.908885 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.908888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.908907 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:40.909020 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.909027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.909756 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.909759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.910586 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.910596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.911301 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.911304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.911399 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.911420 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.911497 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.911584 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":76,"threads_started":1}
I20260929 18:42:40.911715 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.908337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.7.157.124:34937)
0929 18:42:40.908468 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.908473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.908581 (+ 108us) server_negotiation.cc:407] Connection header received
0929 18:42:40.908730 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.908733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.908785 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.908864 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:40.909131 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.909648 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.910711 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.911179 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.911270 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.911429 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.911520 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.911586 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:42:40.912405 14724 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 18:42:40.912652 14753 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3bffeef2ce1474981574d052413f6b6. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } } }
I20260929 18:42:40.912794 14753 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: This master's current role is: LEARNER
W20260929 18:42:40.919790 15043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:40.919790 15040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:40.919791 15041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:40.920019 15034 server_base.cc:1034] running on GCE node
I20260929 18:42:40.920522 15034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:40.921152 15034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:40.922302 15034 hybrid_clock.cc:648] HybridClock initialized: now 1790707360922281 us; error 31 us; skew 500 ppm
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:40.924857 15034 init.cc:377] Logged in from keytab as oryx/127.7.157.67@KRBTEST.COM (short username oryx)
I20260929 18:42:40.925735 15034 webserver.cc:460] Webserver started at http://127.7.157.67:46613/ using document root <none> and password file <none>
I20260929 18:42:40.926276 15034 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:40.926355 15034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:40.930164 15034 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:40.932332 15050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:40.933604 15034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:40.933699 15034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "e28610da940c495f9a268825f9994a3d"
format_stamp: "Formatted at 2026-09-29 18:42:39 on dist-test-slave-smpn"
server_key: "ba4e8920759584402c4654c47a8c8a59"
server_key_iv: "9087c88e5aa2edfc3be8fca2c8f6ff59"
server_key_version: "encryptionkey@0"
I20260929 18:42:40.934245 15034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:40.950865 15034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:40.954227 15034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:40.954428 15034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:40.955034 15034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:40.955938 15034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:40.955998 15034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:40.956055 15034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:40.956086 15034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:40.965212 15034 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:37959
I20260929 18:42:40.965260 15163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:37959 every 8 connection(s)
I20260929 18:42:40.966185 15034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:42:40.973368 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 15034
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.124@KRBTEST.COM
Sep 29 18:42:40 dist-test-slave-smpn krb5kdc[13874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707360, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.125@KRBTEST.COM
I20260929 18:42:40.985028 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.967883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:39449 (local address 127.7.157.125:42879)
0929 18:42:40.968067 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.968072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.969025 (+ 953us) server_negotiation.cc:407] Connection header received
0929 18:42:40.970083 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.970087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.970142 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.970212 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.971553 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.972357 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.973714 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.974026 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.974091 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.978060 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.978064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.978067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.978094 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.982825 (+ 4731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.983796 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.983799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.983801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.983882 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.984145 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.984148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.984149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.984335 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.984409 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.984772 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.984876 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:42:40.985739 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.968349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:34937 (local address 127.7.157.67:54599)
0929 18:42:40.968800 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.968848 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.969898 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.970384 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.970392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.970835 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.971435 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.971440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.972462 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.972465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.973453 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.973463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.974154 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.974158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.974787 (+ 629us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.975202 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.975213 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.977918 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.983836 (+ 5918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.983839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.983841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.984077 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.984371 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.984374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.984376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.984456 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.984873 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.984876 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.984962 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.985070 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.985195 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":626,"spinlock_wait_cycles":22016,"thread_start_us":112,"threads_started":1}
I20260929 18:42:40.985977 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.969301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.67:45669)
0929 18:42:40.969523 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.969536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.969898 (+ 362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.970384 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.970392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.970843 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.971431 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.971440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.972212 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.972224 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.973457 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.973466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.974154 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.974157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.974462 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.975002 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.975021 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.979325 (+ 4304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.981376 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.981382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.981400 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.981948 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.983145 (+ 1197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.983148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.983149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.983343 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.983800 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.983808 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.984131 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.984348 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.984706 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":203,"spinlock_wait_cycles":15872,"thread_start_us":76,"threads_started":1}
I20260929 18:42:40.986436 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.969418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:45669 (local address 127.7.157.126:45147)
0929 18:42:40.969530 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.969535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.969594 (+ 59us) server_negotiation.cc:407] Connection header received
0929 18:42:40.970098 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.970101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.970154 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.970204 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.971571 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.972082 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.973581 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.973894 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.973961 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.979465 (+ 5504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.979468 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.979471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.979495 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.981245 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.982932 (+ 1687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.982935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.982937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.983026 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.983438 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.983441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.983443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.983658 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.983751 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.986179 (+ 2428us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.986303 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:42:40.986768 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.968350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42879 (local address 127.7.157.67:39449)
0929 18:42:40.968906 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:40.968921 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:40.969892 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:40.970383 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:40.970390 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:40.970833 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:40.971419 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.971432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.972844 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.972847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.973610 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:40.973617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.974232 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.974235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:40.974790 (+ 555us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.975557 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:40.975571 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:40.977877 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:40.983473 (+ 5596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.983476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.983478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.983686 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.983970 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:40.983973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:40.983974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:40.984052 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:40.984446 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:40.984448 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:40.984528 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 18:42:40.984622 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 18:42:40.986617 (+ 1995us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":2365,"spinlock_wait_cycles":32256,"thread_start_us":119,"threads_started":1}
I20260929 18:42:40.986801 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.967872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:54599 (local address 127.7.157.124:34937)
0929 18:42:40.968009 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.968013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.968942 (+ 929us) server_negotiation.cc:407] Connection header received
0929 18:42:40.970098 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.970101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.970157 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.970204 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.971579 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.972357 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.973595 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.973889 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.973961 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.978054 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.978058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.978061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.978085 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.983750 (+ 5665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.984185 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.984188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.984190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.984272 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.984553 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.984557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.984558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.984749 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.984833 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.986477 (+ 1644us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.986655 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":54,"spinlock_wait_cycles":1920}
I20260929 18:42:40.987131 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.977527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37720 (local address 127.7.157.124:34937)
0929 18:42:40.977727 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.977731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.977743 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:42:40.977777 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.977780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.977822 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.977904 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.978905 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.979647 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.980714 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.981023 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.981109 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.982732 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.982735 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.982738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.982764 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.984847 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.985238 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.985242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.985243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.985325 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.985598 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.985601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.985602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.985804 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.985939 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.986451 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.986971 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":59,"threads_started":1}
I20260929 18:42:40.987950 15164 heartbeater.cc:347] Connected to a master server at 127.7.157.124:34937
I20260929 18:42:40.988183 15164 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:40.988785 15164 heartbeater.cc:511] Master 127.7.157.124:34937 requested a full tablet report, sending...
I20260929 18:42:40.989486 15165 heartbeater.cc:347] Connected to a master server at 127.7.157.126:45147
I20260929 18:42:40.989565 15165 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:40.989713 15165 heartbeater.cc:511] Master 127.7.157.126:45147 requested a full tablet report, sending...
I20260929 18:42:40.989955 14705 ts_manager.cc:194] Registered new tserver with Master: e28610da940c495f9a268825f9994a3d (127.7.157.67:37959)
I20260929 18:42:40.990157 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.975839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.7.157.126:45147)
0929 18:42:40.976057 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.976061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.976695 (+ 634us) server_negotiation.cc:407] Connection header received
0929 18:42:40.976749 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.976752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.976801 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.976881 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.977663 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.978402 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.979610 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.979949 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.980042 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.981552 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.981555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.981558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.981581 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.983194 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.986705 (+ 3511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.986709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.986711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.986791 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.988985 (+ 2194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.988989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.988990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.989177 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.989293 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.989880 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.990008 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":64,"threads_started":1}
I20260929 18:42:40.990260 15166 heartbeater.cc:347] Connected to a master server at 127.7.157.125:42879
I20260929 18:42:40.990324 15166 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:40.990345 13919 ts_manager.cc:194] Re-registered known tserver with Master: e28610da940c495f9a268825f9994a3d (127.7.157.67:37959)
I20260929 18:42:40.990473 15166 heartbeater.cc:511] Master 127.7.157.125:42879 requested a full tablet report, sending...
I20260929 18:42:40.990955 13919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:45669
I20260929 18:42:40.991266 14520 ts_manager.cc:194] Registered new tserver with Master: e28610da940c495f9a268825f9994a3d (127.7.157.67:37959)
I20260929 18:42:40.992060 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:40.976326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.7.157.125:42879)
0929 18:42:40.979856 (+ 3530us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:40.979860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:40.979871 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:42:40.979902 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:40.979905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:40.979945 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:40.980783 (+ 838us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:40.980877 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.981564 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.982602 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:40.982924 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:40.983001 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:40.987803 (+ 4802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:40.987807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:40.987809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:40.987833 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:40.989374 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.990460 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.990464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.990465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.990545 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:40.990833 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:40.990837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:40.990839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:40.991039 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:40.991721 (+ 682us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:40.991794 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:42:40.991912 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3460,"thread_start_us":78,"threads_started":1}
I20260929 18:42:40.993944 13919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:42:40.996941 13919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:40.999312 14675 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: NON_VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: true } } }
I20260929 18:42:40.999442 14675 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:41.001077 14675 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6: attempting to promote NON_VOTER fa02076e43f842069ef4502c6855b606 to VOTER
I20260929 18:42:41.001428 14675 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } }
I20260929 18:42:41.002442 14724 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 LEARNER]: Refusing update from remote peer d3bffeef2ce1474981574d052413f6b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 18:42:41.002441 14522 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 FOLLOWER]: Refusing update from remote peer d3bffeef2ce1474981574d052413f6b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 18:42:41.002789 15182 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 18:42:41.002925 14675 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 18:42:41.004624 14675 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fa02076e43f842069ef4502c6855b606 (127.7.157.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } } }
I20260929 18:42:41.005002 14724 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fa02076e43f842069ef4502c6855b606 (127.7.157.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } } }
I20260929 18:42:41.006047 14675 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } } }
I20260929 18:42:41.006155 14675 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:41.006259 15183 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } } }
I20260929 18:42:41.006359 15183 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:41.006613 14753 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } } }
I20260929 18:42:41.006724 14753 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:42:41.007406 14522 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fa02076e43f842069ef4502c6855b606 (127.7.157.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } } }
I20260929 18:42:41.006271 14675 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } } }
I20260929 18:42:41.008769 14675 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:41.009766 15192 sys_catalog.cc:455] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3bffeef2ce1474981574d052413f6b6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } } }
I20260929 18:42:41.009891 15192 sys_catalog.cc:458] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:42:41.019166 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:41.014027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:42467 (local address 127.0.0.1:41246)
0929 18:42:41.014179 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:41.014194 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:41.014277 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:41.015033 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:41.015036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:41.015053 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:41.015185 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:41.015191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.016740 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.016743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:41.017950 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:41.017959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.018769 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.018773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:41.018852 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:41.018873 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:42:41.018933 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:42:41.019000 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":56}
I20260929 18:42:41.019207 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:41.013900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41246 (local address 127.7.157.66:42467)
0929 18:42:41.014502 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:41.014514 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:41.014538 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:42:41.014618 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:41.014623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:41.014873 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:41.015350 (+ 477us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:41.015484 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.016623 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.018072 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.018647 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.018771 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:41.018922 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:41.018971 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 18:42:41.019058 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":107,"threads_started":1}
I20260929 18:42:41.019539 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:41.014133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:37959 (local address 127.0.0.1:46966)
0929 18:42:41.014377 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:41.015022 (+ 645us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:41.015150 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:41.015644 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:41.015647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:41.015666 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:41.015772 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:41.015780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.016981 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.016985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:41.018073 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:41.018082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.019186 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.019190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:41.019280 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:41.019303 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:42:41.019350 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 18:42:41.019410 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":21,"thread_start_us":107,"threads_started":1}
I20260929 18:42:41.020380 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:41.014442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:45901 (local address 127.0.0.1:38582)
0929 18:42:41.015742 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:41.015754 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:41.015834 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:41.016232 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:41.016235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:41.016252 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:41.016342 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:41.016349 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.017598 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.017601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:41.018765 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:41.018775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.019988 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.019991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:41.020075 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:41.020095 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:42:41.020182 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:42:41.020258 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1248,"mutex_wait_us":4,"thread_start_us":81,"threads_started":1}
I20260929 18:42:41.020614 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:41.014598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38582 (local address 127.7.157.65:45901)
0929 18:42:41.015617 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:41.015628 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:41.015918 (+ 290us) server_negotiation.cc:407] Connection header received
0929 18:42:41.015982 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:41.015986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:41.016122 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:41.016248 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:41.016427 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.017477 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.018896 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.019739 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.019828 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:41.019932 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:41.020238 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 18:42:41.020329 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":100,"threads_started":1}
I20260929 18:42:41.022145 14961 tablet_service.cc:1467] Processing CreateTablet for tablet 3cbd2da83b024216b9dd1f6591b4befc (DEFAULT_TABLE table=test-table [id=2f1e9d7bf8d34b84a6fa0f5326453747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:41.023087 14961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cbd2da83b024216b9dd1f6591b4befc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:41.020133 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:41.014567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46966 (local address 127.7.157.67:37959)
0929 18:42:41.014988 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:41.014999 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:41.015106 (+ 107us) server_negotiation.cc:407] Connection header received
0929 18:42:41.015288 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:41.015294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:41.015497 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:41.015653 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:41.015861 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.016868 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.018190 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.019003 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.019704 (+ 701us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:41.019859 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:41.019909 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 18:42:41.019998 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":108,"threads_started":1}
I20260929 18:42:41.022876 14824 tablet_service.cc:1467] Processing CreateTablet for tablet 3cbd2da83b024216b9dd1f6591b4befc (DEFAULT_TABLE table=test-table [id=2f1e9d7bf8d34b84a6fa0f5326453747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:41.023770 14824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cbd2da83b024216b9dd1f6591b4befc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:41.029974 15198 tablet_bootstrap.cc:492] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff: Bootstrap starting.
I20260929 18:42:41.031199 15199 tablet_bootstrap.cc:492] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a: Bootstrap starting.
I20260929 18:42:41.032091 15198 tablet_bootstrap.cc:654] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:41.031790 15098 tablet_service.cc:1467] Processing CreateTablet for tablet 3cbd2da83b024216b9dd1f6591b4befc (DEFAULT_TABLE table=test-table [id=2f1e9d7bf8d34b84a6fa0f5326453747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:41.032622 15098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cbd2da83b024216b9dd1f6591b4befc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:41.032888 15198 log.cc:824] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:41.033499 15199 tablet_bootstrap.cc:654] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:41.034262 15199 log.cc:824] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:41.034862 15198 tablet_bootstrap.cc:492] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff: No bootstrap required, opened a new log
I20260929 18:42:41.035032 15198 ts_tablet_manager.cc:1397] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:42:41.038404 15198 raft_consensus.cc:348] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } }
I20260929 18:42:41.038621 15198 raft_consensus.cc:374] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:41.038679 15198 raft_consensus.cc:729] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec3bc489671458189f218079f60c6ff, State: Initialized, Role: FOLLOWER
I20260929 18:42:41.039124 15198 consensus_queue.cc:260] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } }
I20260929 18:42:41.039952 15198 ts_tablet_manager.cc:1428] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260929 18:42:41.040402 15028 heartbeater.cc:503] Master 127.7.157.126:45147 was elected leader, sending a full tablet report...
I20260929 18:42:41.041591 15199 tablet_bootstrap.cc:492] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a: No bootstrap required, opened a new log
I20260929 18:42:41.041778 15199 ts_tablet_manager.cc:1397] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260929 18:42:41.046588 15199 raft_consensus.cc:348] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } }
I20260929 18:42:41.046846 15199 raft_consensus.cc:374] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:41.046892 15199 raft_consensus.cc:729] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8d904fb2c54e72bbf064ec58d2a15a, State: Initialized, Role: FOLLOWER
I20260929 18:42:41.047333 15199 consensus_queue.cc:260] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } }
I20260929 18:42:41.047698 15202 tablet_bootstrap.cc:492] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d: Bootstrap starting.
I20260929 18:42:41.048282 15199 ts_tablet_manager.cc:1428] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260929 18:42:41.050302 14891 heartbeater.cc:503] Master 127.7.157.126:45147 was elected leader, sending a full tablet report...
I20260929 18:42:41.050535 15202 tablet_bootstrap.cc:654] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:41.051578 15202 log.cc:824] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:41.053855 15202 tablet_bootstrap.cc:492] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d: No bootstrap required, opened a new log
I20260929 18:42:41.054035 15202 ts_tablet_manager.cc:1397] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:42:41.057353 15202 raft_consensus.cc:348] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } }
I20260929 18:42:41.057566 15202 raft_consensus.cc:374] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:41.057619 15202 raft_consensus.cc:729] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28610da940c495f9a268825f9994a3d, State: Initialized, Role: FOLLOWER
I20260929 18:42:41.058012 15202 consensus_queue.cc:260] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } }
I20260929 18:42:41.058725 15165 heartbeater.cc:503] Master 127.7.157.126:45147 was elected leader, sending a full tablet report...
I20260929 18:42:41.058929 15202 ts_tablet_manager.cc:1428] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260929 18:42:41.090762 15204 raft_consensus.cc:482] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:41.090961 15204 raft_consensus.cc:504] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } }
W20260929 18:42:41.091475 14893 tablet.cc:2369] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:42:41.092293 15204 leader_election.cc:288] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ec3bc489671458189f218079f60c6ff (127.7.157.66:42467), e28610da940c495f9a268825f9994a3d (127.7.157.67:37959)
I20260929 18:42:41.096647 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:41.092661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:58127 (local address 127.7.157.67:37959)
0929 18:42:41.092801 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:41.092805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:41.092969 (+ 164us) server_negotiation.cc:407] Connection header received
0929 18:42:41.093081 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:41.093083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:41.093120 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:41.093179 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:41.093413 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.094113 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.095472 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.096078 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.096183 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:41.096235 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:41.096451 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 18:42:41.096520 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:42:41.096647 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:41.092571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:49341 (local address 127.7.157.66:42467)
0929 18:42:41.092694 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:41.092699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:41.092716 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:42:41.092825 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:41.092828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:41.092870 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:41.092931 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:41.093200 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.093975 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.095477 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.096059 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.096183 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:41.096236 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:41.096451 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 18:42:41.096528 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:42:41.096664 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:41.092515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:37959 (local address 127.7.157.65:58127)
0929 18:42:41.092898 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:41.092911 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:41.092986 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:41.093199 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:41.093202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:41.093226 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:41.093319 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:41.093324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.094192 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.094195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:41.095371 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:41.095379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.096175 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.096178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:41.096317 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:41.096371 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 18:42:41.096431 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:42:41.096513 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":353,"spinlock_wait_cycles":2816}
I20260929 18:42:41.096673 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:41.092501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:42467 (local address 127.7.157.65:49341)
0929 18:42:41.092614 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:41.092628 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:41.092711 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:41.092948 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:41.092951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:41.092971 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:41.093102 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:41.093107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.094078 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.094080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:41.095368 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:41.095379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.096175 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.096178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:41.096320 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:41.096358 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 18:42:41.096431 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:42:41.096513 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":43}
I20260929 18:42:41.097277 15118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cbd2da83b024216b9dd1f6591b4befc" candidate_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28610da940c495f9a268825f9994a3d" is_pre_election: true
I20260929 18:42:41.097496 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cbd2da83b024216b9dd1f6591b4befc" candidate_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ec3bc489671458189f218079f60c6ff" is_pre_election: true
I20260929 18:42:41.097599 15118 raft_consensus.cc:2454] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a8d904fb2c54e72bbf064ec58d2a15a in term 0.
I20260929 18:42:41.097759 14981 raft_consensus.cc:2454] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a8d904fb2c54e72bbf064ec58d2a15a in term 0.
I20260929 18:42:41.098063 14779 leader_election.cc:302] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a8d904fb2c54e72bbf064ec58d2a15a, e28610da940c495f9a268825f9994a3d; no voters:
I20260929 18:42:41.098363 15204 raft_consensus.cc:2781] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:41.098439 15204 raft_consensus.cc:482] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:41.098469 15204 raft_consensus.cc:3037] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:41.099527 15204 raft_consensus.cc:504] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } }
I20260929 18:42:41.099872 15204 leader_election.cc:288] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [CANDIDATE]: Term 1 election: Requested vote from peers 4ec3bc489671458189f218079f60c6ff (127.7.157.66:42467), e28610da940c495f9a268825f9994a3d (127.7.157.67:37959)
I20260929 18:42:41.100178 15118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cbd2da83b024216b9dd1f6591b4befc" candidate_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28610da940c495f9a268825f9994a3d"
I20260929 18:42:41.100188 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cbd2da83b024216b9dd1f6591b4befc" candidate_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ec3bc489671458189f218079f60c6ff"
I20260929 18:42:41.100303 14981 raft_consensus.cc:3037] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:41.100303 15118 raft_consensus.cc:3037] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:41.101542 15118 raft_consensus.cc:2454] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a8d904fb2c54e72bbf064ec58d2a15a in term 1.
I20260929 18:42:41.101542 14981 raft_consensus.cc:2454] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a8d904fb2c54e72bbf064ec58d2a15a in term 1.
I20260929 18:42:41.101814 14778 leader_election.cc:302] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4ec3bc489671458189f218079f60c6ff, 5a8d904fb2c54e72bbf064ec58d2a15a, e28610da940c495f9a268825f9994a3d; no voters:
I20260929 18:42:41.101972 15204 raft_consensus.cc:2781] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:41.102169 15204 raft_consensus.cc:686] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [term 1 LEADER]: Becoming Leader. State: Replica: 5a8d904fb2c54e72bbf064ec58d2a15a, State: Running, Role: LEADER
I20260929 18:42:41.102515 15204 consensus_queue.cc:237] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } }
I20260929 18:42:41.105839 13919 catalog_manager.cc:5495] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a8d904fb2c54e72bbf064ec58d2a15a (127.7.157.65). New cstate: current_term: 1 leader_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } health_report { overall_health: HEALTHY } } }
I20260929 18:42:41.140352 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:41.136501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.7.157.65:45901)
0929 18:42:41.136606 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:41.136610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:41.136624 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:42:41.136759 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:41.136761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:41.136812 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:41.136913 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:41.137118 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.137603 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.138837 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.139049 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.139166 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:41.139511 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:41.139939 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:41.140055 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:41.140138 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 18:42:41.140197 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 18:42:41.146389 15118 raft_consensus.cc:1261] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d [term 1 FOLLOWER]: Refusing update from remote peer 5a8d904fb2c54e72bbf064ec58d2a15a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:41.146389 14981 raft_consensus.cc:1261] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff [term 1 FOLLOWER]: Refusing update from remote peer 5a8d904fb2c54e72bbf064ec58d2a15a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:41.147179 15207 consensus_queue.cc:1035] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:41.147393 15204 consensus_queue.cc:1035] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:41.156003 15210 mvcc.cc:205] Tried to move back new op lower bound from 7334737351247990784 to 7334737351078477824. Current Snapshot: MvccSnapshot[applied={T|T < 7334737351247990784}]
I20260929 18:42:41.156612 15211 mvcc.cc:205] Tried to move back new op lower bound from 7334737351247990784 to 7334737351078477824. Current Snapshot: MvccSnapshot[applied={T|T < 7334737351247990784}]
I20260929 18:42:41.157294 15212 mvcc.cc:205] Tried to move back new op lower bound from 7334737351247990784 to 7334737351078477824. Current Snapshot: MvccSnapshot[applied={T|T < 7334737351247990784}]
I20260929 18:42:41.166069 13919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:41.166224 13919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:41.171025 13919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3bffeef2ce1474981574d052413f6b6: Sending DeleteTablet for 3 replicas of tablet 3cbd2da83b024216b9dd1f6591b4befc
I20260929 18:42:41.171691 14961 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cbd2da83b024216b9dd1f6591b4befc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:41 UTC) from {username='oryx', principal='oryx/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:41246
I20260929 18:42:41.172030 15098 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cbd2da83b024216b9dd1f6591b4befc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:41 UTC) from {username='oryx', principal='oryx/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:46966
I20260929 18:42:41.172544 15221 tablet_replica.cc:330] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d: stopping tablet replica
I20260929 18:42:41.172633 14824 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cbd2da83b024216b9dd1f6591b4befc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:41 UTC) from {username='oryx', principal='oryx/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:38582
I20260929 18:42:41.173012 15221 raft_consensus.cc:2229] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:41.173053 15222 tablet_replica.cc:330] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a: stopping tablet replica
I20260929 18:42:41.173341 15222 raft_consensus.cc:2229] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:42:41.173530 15221 raft_consensus.cc:2258] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:41.173676 15222 raft_consensus.cc:2258] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:41.174711 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 13887
I20260929 18:42:41.175673 15222 ts_tablet_manager.cc:1905] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:41.176214 15221 ts_tablet_manager.cc:1905] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:41.178282 15222 ts_tablet_manager.cc:1918] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:41.178373 15222 log.cc:1196] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-0/wal/wals/3cbd2da83b024216b9dd1f6591b4befc
I20260929 18:42:41.178670 15222 ts_tablet_manager.cc:1939] T 3cbd2da83b024216b9dd1f6591b4befc P 5a8d904fb2c54e72bbf064ec58d2a15a: Deleting consensus metadata
I20260929 18:42:41.178864 15221 ts_tablet_manager.cc:1918] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:41.179003 15221 log.cc:1196] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/3cbd2da83b024216b9dd1f6591b4befc
I20260929 18:42:41.179388 15221 ts_tablet_manager.cc:1939] T 3cbd2da83b024216b9dd1f6591b4befc P e28610da940c495f9a268825f9994a3d: Deleting consensus metadata
W20260929 18:42:41.181365 14779 connection.cc:535] server connection from 127.0.0.1:38582 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 18:42:41.181412 15052 connection.cc:535] server connection from 127.0.0.1:46966 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:42:41.182765 15220 tablet_replica.cc:330] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff: stopping tablet replica
I20260929 18:42:41.183117 15220 raft_consensus.cc:2229] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:41.183439 15220 raft_consensus.cc:2258] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:41.185684 15220 ts_tablet_manager.cc:1905] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:41.188282 15220 ts_tablet_manager.cc:1918] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:41.188433 15220 log.cc:1196] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/3cbd2da83b024216b9dd1f6591b4befc
I20260929 18:42:41.188799 15220 ts_tablet_manager.cc:1939] T 3cbd2da83b024216b9dd1f6591b4befc P 4ec3bc489671458189f218079f60c6ff: Deleting consensus metadata
W20260929 18:42:41.189966 14916 connection.cc:441] server connection from 127.0.0.1:41246 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41246 (request call id 1) could send its response
I20260929 18:42:41.194511 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:41.174093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.7.157.125:42879)
0929 18:42:41.174332 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:41.174336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:41.184027 (+ 9691us) server_negotiation.cc:407] Connection header received
0929 18:42:41.184092 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:41.184095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:41.184150 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:41.184221 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:41.185852 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.186571 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.187698 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.188040 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.188120 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:41.189892 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:41.189896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:41.189898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:41.189923 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:41.191805 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:41.192704 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:41.192708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:41.192710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:41.192792 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:41.193321 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:41.193324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:41.193326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:41.193553 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:41.193658 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:41.193981 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 18:42:41.194118 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"spinlock_wait_cycles":3328}
I20260929 18:42:41.194993 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:41.184540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37732 (local address 127.7.157.124:34937)
0929 18:42:41.184643 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:41.184647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:41.185292 (+ 645us) server_negotiation.cc:407] Connection header received
0929 18:42:41.185399 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:41.185402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:41.185450 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:41.185523 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:41.186731 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.187546 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.189236 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.189574 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.189646 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:41.191740 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:41.191744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:41.191747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:41.191775 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:41.193295 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:41.193734 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:41.193738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:41.193740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:41.193819 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:41.194099 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:41.194103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:41.194105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:41.194317 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:41.194415 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:41.194734 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 18:42:41.194847 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:42:41.197345 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:41.184540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.7.157.125:42879)
0929 18:42:41.184642 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:41.184646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:41.185561 (+ 915us) server_negotiation.cc:407] Connection header received
0929 18:42:41.188836 (+ 3275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:41.188840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:41.188882 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:41.188928 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:41.190336 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.191098 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.192154 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.192463 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.192532 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:41.193919 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:41.193923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:41.193925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:41.193960 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:41.195492 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:41.196098 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:41.196101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:41.196103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:41.196181 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:41.196472 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:41.196475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:41.196477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:41.196673 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:41.196769 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:41.197081 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 18:42:41.197187 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39,"spinlock_wait_cycles":5248}
I20260929 18:42:41.203050 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:41.174193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.7.157.124:34937)
0929 18:42:41.174419 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:41.174423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:41.195931 (+ 21508us) server_negotiation.cc:407] Connection header received
0929 18:42:41.196089 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:41.196092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:41.196131 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:41.196178 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:41.196863 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.197427 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.198338 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:41.198579 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:41.198660 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:41.199897 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:41.199901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:41.199904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:41.199928 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:41.201152 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:41.201521 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:41.201524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:41.201526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:41.201600 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:41.201839 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:41.201843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:41.201845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:41.202044 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:41.202168 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:41.202774 (+ 606us) server_negotiation.cc:299] Negotiation successful
0929 18:42:41.202893 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260929 18:42:42.161566 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:42.161068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.67:32973)
0929 18:42:42.161346 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:42.161457 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
I20260929 18:42:42.161878 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:42.161264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.66:35399)
0929 18:42:42.161485 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:42.161576 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
W20260929 18:42:42.161893 15165 heartbeater.cc:644] Failed to heartbeat to 127.7.157.126:45147 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
W20260929 18:42:42.162204 15028 heartbeater.cc:644] Failed to heartbeat to 127.7.157.126:45147 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
I20260929 18:42:42.162479 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:42.162074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.65:43657)
0929 18:42:42.162322 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:42.162400 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
W20260929 18:42:42.162698 14891 heartbeater.cc:644] Failed to heartbeat to 127.7.157.126:45147 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
I20260929 18:42:42.671207 15234 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3bffeef2ce1474981574d052413f6b6)
I20260929 18:42:42.671355 15235 raft_consensus.cc:482] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3bffeef2ce1474981574d052413f6b6)
I20260929 18:42:42.671466 15234 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } }
I20260929 18:42:42.671542 15235 raft_consensus.cc:504] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } }
I20260929 18:42:42.672504 15234 leader_election.cc:288] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3bffeef2ce1474981574d052413f6b6 (127.7.157.126:45147), e697954f7ed54a67a0a9a51adfc5e1ba (127.7.157.125:42879)
I20260929 18:42:42.672538 15235 leader_election.cc:288] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3bffeef2ce1474981574d052413f6b6 (127.7.157.126:45147), fa02076e43f842069ef4502c6855b606 (127.7.157.124:34937)
I20260929 18:42:42.673188 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:42.672790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:52416)
0929 18:42:42.673018 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:42.673088 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":51,"threads_started":1}
I20260929 18:42:42.673190 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:42.672678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:52408)
0929 18:42:42.673008 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:42.673119 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":85,"threads_started":1}
W20260929 18:42:42.673445 14697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
W20260929 18:42:42.673472 14506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
I20260929 18:42:42.674295 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:42.674027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:52426)
0929 18:42:42.674172 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:42.674225 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260929 18:42:42.674381 14697 leader_election.cc:334] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d3bffeef2ce1474981574d052413f6b6 (127.7.157.126:45147): Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
I20260929 18:42:42.674643 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:42.674269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:52428)
0929 18:42:42.674531 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:42.674581 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260929 18:42:42.674738 14506 leader_election.cc:334] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d3bffeef2ce1474981574d052413f6b6 (127.7.157.126:45147): Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
I20260929 18:42:42.676906 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:42.673061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42879 (local address 127.0.0.1:37480)
0929 18:42:42.673325 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:42.673337 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:42.673433 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:42.673862 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:42.673865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:42.673884 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:42.673999 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:42.674005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:42.675101 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:42.675104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:42.675901 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:42.675910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:42.676527 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:42.676530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:42.676636 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:42.676654 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:42:42.676719 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 18:42:42.676788 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":60,"threads_started":1}
I20260929 18:42:42.676924 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:42.673108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.7.157.125:42879)
0929 18:42:42.673621 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:42.673626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:42.673638 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:42:42.673685 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:42.673688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:42.673740 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:42.674101 (+ 361us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:42.674209 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:42.674973 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:42.676013 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:42.676423 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:42.676512 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:42.676568 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:42.676750 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 18:42:42.676819 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":446,"spinlock_wait_cycles":4480,"thread_start_us":58,"threads_started":1}
I20260929 18:42:42.677058 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:42.673260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37742 (local address 127.7.157.124:34937)
0929 18:42:42.673884 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:42.673889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:42.673902 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:42:42.673943 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:42.673946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:42.673988 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:42.674357 (+ 369us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:42.674446 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:42.675177 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:42.676187 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:42.676596 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:42.676684 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:42.676734 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:42.676890 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 18:42:42.676952 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":566,"thread_start_us":45,"threads_started":1}
I20260929 18:42:42.677059 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:42.672765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:34937 (local address 127.0.0.1:37742)
0929 18:42:42.673307 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:42.673322 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:42.673424 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:42.674107 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:42.674110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:42.674130 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:42.674255 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:42.674262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:42.675277 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:42.675280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:42.676070 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:42.676084 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:42.676697 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:42.676700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:42.676790 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:42.676807 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:42:42.676868 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 18:42:42.676937 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":20,"thread_start_us":58,"threads_started":1}
I20260929 18:42:42.677385 14530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa02076e43f842069ef4502c6855b606" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" is_pre_election: true
I20260929 18:42:42.677508 14724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fa02076e43f842069ef4502c6855b606" is_pre_election: true
I20260929 18:42:42.677597 14530 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa02076e43f842069ef4502c6855b606 in term 1.
I20260929 18:42:42.677657 14724 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e697954f7ed54a67a0a9a51adfc5e1ba in term 1.
I20260929 18:42:42.677939 14504 leader_election.cc:302] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e697954f7ed54a67a0a9a51adfc5e1ba, fa02076e43f842069ef4502c6855b606; no voters: d3bffeef2ce1474981574d052413f6b6
I20260929 18:42:42.677942 14697 leader_election.cc:302] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e697954f7ed54a67a0a9a51adfc5e1ba, fa02076e43f842069ef4502c6855b606; no voters: d3bffeef2ce1474981574d052413f6b6
I20260929 18:42:42.678120 15235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 18:42:42.678174 15234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 18:42:42.678251 15234 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3bffeef2ce1474981574d052413f6b6)
I20260929 18:42:42.678268 15235 raft_consensus.cc:482] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3bffeef2ce1474981574d052413f6b6)
I20260929 18:42:42.678301 15234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:42:42.678323 15235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:42:42.679391 15234 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } }
I20260929 18:42:42.679404 15235 raft_consensus.cc:504] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } }
I20260929 18:42:42.679708 15234 leader_election.cc:288] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [CANDIDATE]: Term 2 election: Requested vote from peers d3bffeef2ce1474981574d052413f6b6 (127.7.157.126:45147), e697954f7ed54a67a0a9a51adfc5e1ba (127.7.157.125:42879)
I20260929 18:42:42.679765 15235 leader_election.cc:288] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [CANDIDATE]: Term 2 election: Requested vote from peers d3bffeef2ce1474981574d052413f6b6 (127.7.157.126:45147), fa02076e43f842069ef4502c6855b606 (127.7.157.124:34937)
I20260929 18:42:42.680027 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:42.679836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:52430)
0929 18:42:42.679919 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:42.679966 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 18:42:42.680069 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:42.679846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:52434)
0929 18:42:42.679961 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:42.680009 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 18:42:42.680379 14530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa02076e43f842069ef4502c6855b606" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba"
I20260929 18:42:42.680387 14724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fa02076e43f842069ef4502c6855b606"
I20260929 18:42:42.680531 14724 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e697954f7ed54a67a0a9a51adfc5e1ba in current term 2: Already voted for candidate fa02076e43f842069ef4502c6855b606 in this term.
I20260929 18:42:42.680532 14530 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fa02076e43f842069ef4502c6855b606 in current term 2: Already voted for candidate e697954f7ed54a67a0a9a51adfc5e1ba in this term.
I20260929 18:42:42.680789 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:42.680466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:52440)
0929 18:42:42.680675 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:42.680735 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260929 18:42:42.680898 14506 leader_election.cc:334] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d3bffeef2ce1474981574d052413f6b6 (127.7.157.126:45147): Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
I20260929 18:42:42.680945 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:42.680473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:52450)
0929 18:42:42.680627 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:42.680889 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 18:42:42.680959 14506 leader_election.cc:302] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e697954f7ed54a67a0a9a51adfc5e1ba; no voters: d3bffeef2ce1474981574d052413f6b6, fa02076e43f842069ef4502c6855b606
W20260929 18:42:42.681011 14697 leader_election.cc:334] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d3bffeef2ce1474981574d052413f6b6 (127.7.157.126:45147): Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
I20260929 18:42:42.681066 14697 leader_election.cc:302] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa02076e43f842069ef4502c6855b606; no voters: d3bffeef2ce1474981574d052413f6b6, e697954f7ed54a67a0a9a51adfc5e1ba
I20260929 18:42:42.681084 15235 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 18:42:42.681181 15234 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 18:42:43.162823 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:43.162418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.67:36097)
0929 18:42:43.162667 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:43.162733 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
I20260929 18:42:43.163057 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:43.162697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.66:51457)
0929 18:42:43.162905 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:43.162966 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":63,"threads_started":1}
I20260929 18:42:43.163516 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:43.163181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.65:35339)
0929 18:42:43.163397 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:43.163444 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20260929 18:42:44.164004 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.163577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.67:48415)
0929 18:42:44.163837 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.163913 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
I20260929 18:42:44.164053 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.163629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.66:48371)
0929 18:42:44.163878 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.163942 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":51,"threads_started":1}
I20260929 18:42:44.164433 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.164074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.65:60641)
0929 18:42:44.164311 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.164360 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":61,"threads_started":1}
I20260929 18:42:44.379312 15250 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:44.379443 15250 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } }
I20260929 18:42:44.379866 15250 leader_election.cc:288] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d3bffeef2ce1474981574d052413f6b6 (127.7.157.126:45147), e697954f7ed54a67a0a9a51adfc5e1ba (127.7.157.125:42879)
I20260929 18:42:44.380415 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.380034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:52704)
0929 18:42:44.380236 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.380327 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":57,"threads_started":1}
I20260929 18:42:44.380604 14530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa02076e43f842069ef4502c6855b606" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" is_pre_election: true
I20260929 18:42:44.380734 14530 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa02076e43f842069ef4502c6855b606 in term 2.
I20260929 18:42:44.381047 14697 leader_election.cc:302] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e697954f7ed54a67a0a9a51adfc5e1ba, fa02076e43f842069ef4502c6855b606; no voters:
I20260929 18:42:44.381197 15250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 18:42:44.381259 15250 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:44.381286 15250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:42:44.381537 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.381312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:52708)
0929 18:42:44.381440 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.381483 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260929 18:42:44.381673 14697 leader_election.cc:334] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d3bffeef2ce1474981574d052413f6b6 (127.7.157.126:45147): Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
I20260929 18:42:44.382413 15250 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } }
I20260929 18:42:44.382730 15250 leader_election.cc:288] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [CANDIDATE]: Term 3 election: Requested vote from peers d3bffeef2ce1474981574d052413f6b6 (127.7.157.126:45147), e697954f7ed54a67a0a9a51adfc5e1ba (127.7.157.125:42879)
I20260929 18:42:44.383031 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.382838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:52724)
0929 18:42:44.382928 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.382979 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 18:42:44.383239 14530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa02076e43f842069ef4502c6855b606" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba"
I20260929 18:42:44.383378 14530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:42:44.383621 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.383450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:52730)
0929 18:42:44.383529 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.383573 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260929 18:42:44.383723 14697 leader_election.cc:334] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d3bffeef2ce1474981574d052413f6b6 (127.7.157.126:45147): Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
I20260929 18:42:44.384442 14530 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa02076e43f842069ef4502c6855b606 in term 3.
I20260929 18:42:44.384703 14697 leader_election.cc:302] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e697954f7ed54a67a0a9a51adfc5e1ba, fa02076e43f842069ef4502c6855b606; no voters: d3bffeef2ce1474981574d052413f6b6
I20260929 18:42:44.384855 15250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:42:44.385018 15250 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [term 3 LEADER]: Becoming Leader. State: Replica: fa02076e43f842069ef4502c6855b606, State: Running, Role: LEADER
I20260929 18:42:44.385267 15250 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } }
I20260929 18:42:44.386251 15253 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa02076e43f842069ef4502c6855b606. Latest consensus state: current_term: 3 leader_uuid: "fa02076e43f842069ef4502c6855b606" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } } }
I20260929 18:42:44.386341 15253 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:44.386584 15255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:44.388612 15255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:44.388967 15255 catalog_manager.cc:1259] Loaded cluster ID: 50298e1848a94311b5e0994750e859f9
I20260929 18:42:44.389006 15255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:44.389497 15255 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:44.389851 15255 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: Loaded TSK: 0
I20260929 18:42:44.390482 15255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:42:44.398305 14707 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:42:44.401397 14707 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:44.404366 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.404125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:52758)
0929 18:42:44.404220 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.404282 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 18:42:44.404632 14530 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [term 3 FOLLOWER]: Refusing update from remote peer fa02076e43f842069ef4502c6855b606: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 18:42:44.404999 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.404801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.0.0.1:52772)
0929 18:42:44.404885 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.404937 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260929 18:42:44.405086 14697 consensus_peers.cc:487] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 -> Peer d3bffeef2ce1474981574d052413f6b6 (127.7.157.126:45147): Couldn't send request to peer d3bffeef2ce1474981574d052413f6b6. Status: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 18:42:44.405099 15250 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 18:42:44.406476 15257 sys_catalog.cc:455] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa02076e43f842069ef4502c6855b606. Latest consensus state: current_term: 3 leader_uuid: "fa02076e43f842069ef4502c6855b606" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } } }
I20260929 18:42:44.406594 15257 sys_catalog.cc:458] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:42:44.407645 15250 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "fa02076e43f842069ef4502c6855b606" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } } }
I20260929 18:42:44.407760 15250 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:44.407928 15257 sys_catalog.cc:455] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "fa02076e43f842069ef4502c6855b606" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3bffeef2ce1474981574d052413f6b6" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45147 } } peers { permanent_uuid: "e697954f7ed54a67a0a9a51adfc5e1ba" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42879 } attrs { promote: false } } peers { permanent_uuid: "fa02076e43f842069ef4502c6855b606" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 34937 } attrs { promote: false } } }
I20260929 18:42:44.408033 15257 sys_catalog.cc:458] T 00000000000000000000000000000000 P e697954f7ed54a67a0a9a51adfc5e1ba [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:42:44.417802 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.413581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:45901 (local address 127.0.0.1:38596)
0929 18:42:44.413934 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.413947 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:44.414031 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:44.414603 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:44.414606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:44.414627 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:44.414710 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:44.414716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:44.415673 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:44.415676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:44.416556 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:44.416566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:44.417364 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:44.417368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:44.417457 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:44.417486 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 18:42:44.417535 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:42:44.417632 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":21,"spinlock_wait_cycles":1408,"thread_start_us":107,"threads_started":1}
I20260929 18:42:44.418027 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.413126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:42467 (local address 127.0.0.1:41252)
0929 18:42:44.413491 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.413505 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:44.413588 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:44.414355 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:44.414358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:44.414372 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:44.414469 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:44.414475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:44.415559 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:44.415562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:44.416791 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:44.416801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:44.417615 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:44.417618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:44.417708 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:44.417735 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 18:42:44.417814 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:42:44.417903 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":1}
I20260929 18:42:44.418154 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.413262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.7.157.66:42467)
0929 18:42:44.413523 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:44.413528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:44.413548 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:42:44.413771 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:44.413774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:44.414257 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:44.414341 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:44.414583 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:44.415452 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:44.416912 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:44.417520 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:44.417623 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:44.417685 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:44.417836 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 18:42:44.418024 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":99,"threads_started":1}
I20260929 18:42:44.418443 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.414183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.7.157.65:45901)
0929 18:42:44.414389 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:44.414392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:44.414404 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:42:44.414445 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:44.414447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:44.414484 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:44.414593 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:44.414821 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:44.415570 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:44.416669 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:44.417225 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:44.418135 (+ 910us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:44.418203 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:44.418242 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 18:42:44.418320 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
I20260929 18:42:44.418606 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.413876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:37959 (local address 127.0.0.1:46982)
0929 18:42:44.414528 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.414540 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:44.414619 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:44.414982 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:44.414985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:44.415000 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:44.415115 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:44.415121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:44.415988 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:44.415991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:44.417057 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:44.417065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:44.417970 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:44.417974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:44.418262 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:44.418289 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 18:42:44.418360 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 18:42:44.418426 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":73,"threads_started":1}
I20260929 18:42:44.418951 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.414002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.7.157.67:37959)
0929 18:42:44.414244 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:44.414249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:44.414808 (+ 559us) server_negotiation.cc:407] Connection header received
0929 18:42:44.414854 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:44.414856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:44.414894 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:44.414956 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:44.415215 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:44.415864 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:44.417173 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:44.417773 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:44.417858 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:44.417917 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:44.418384 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 18:42:44.418464 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":96,"threads_started":1}
I20260929 18:42:44.418919 14961 tablet_service.cc:1467] Processing CreateTablet for tablet 00ea18bbb9e14da4b0f8d0e2acfe2392 (DEFAULT_TABLE table=test-table [id=8f1b1226812b4c0d94d8548b44450dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:44.419051 14824 tablet_service.cc:1467] Processing CreateTablet for tablet 00ea18bbb9e14da4b0f8d0e2acfe2392 (DEFAULT_TABLE table=test-table [id=8f1b1226812b4c0d94d8548b44450dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:44.419292 14961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ea18bbb9e14da4b0f8d0e2acfe2392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:44.419358 14824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ea18bbb9e14da4b0f8d0e2acfe2392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:44.419404 15098 tablet_service.cc:1467] Processing CreateTablet for tablet 00ea18bbb9e14da4b0f8d0e2acfe2392 (DEFAULT_TABLE table=test-table [id=8f1b1226812b4c0d94d8548b44450dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:44.419652 15098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ea18bbb9e14da4b0f8d0e2acfe2392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:44.421674 15268 tablet_bootstrap.cc:492] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a: Bootstrap starting.
I20260929 18:42:44.421818 15269 tablet_bootstrap.cc:492] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff: Bootstrap starting.
I20260929 18:42:44.422714 15268 tablet_bootstrap.cc:654] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:44.422715 15269 tablet_bootstrap.cc:654] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:44.423285 15270 tablet_bootstrap.cc:492] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d: Bootstrap starting.
I20260929 18:42:44.423877 15268 tablet_bootstrap.cc:492] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a: No bootstrap required, opened a new log
I20260929 18:42:44.423971 15268 ts_tablet_manager.cc:1397] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:44.423997 15270 tablet_bootstrap.cc:654] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:44.424393 15269 tablet_bootstrap.cc:492] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff: No bootstrap required, opened a new log
I20260929 18:42:44.424393 15268 raft_consensus.cc:348] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } }
I20260929 18:42:44.424515 15269 ts_tablet_manager.cc:1397] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:42:44.424515 15268 raft_consensus.cc:374] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:44.424551 15268 raft_consensus.cc:729] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8d904fb2c54e72bbf064ec58d2a15a, State: Initialized, Role: FOLLOWER
I20260929 18:42:44.424677 15268 consensus_queue.cc:260] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } }
I20260929 18:42:44.425055 15268 ts_tablet_manager.cc:1428] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:44.424990 15269 raft_consensus.cc:348] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } }
I20260929 18:42:44.425091 15269 raft_consensus.cc:374] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:44.425119 15269 raft_consensus.cc:729] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec3bc489671458189f218079f60c6ff, State: Initialized, Role: FOLLOWER
I20260929 18:42:44.425556 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.425341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.65:56803)
0929 18:42:44.425438 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.425495 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 18:42:44.425889 15269 consensus_queue.cc:260] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } }
I20260929 18:42:44.426257 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.426021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.65:42717)
0929 18:42:44.426150 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.426195 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 18:42:44.426734 15269 ts_tablet_manager.cc:1428] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:44.427855 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.427604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.66:49965)
0929 18:42:44.427718 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.427774 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 18:42:44.427997 14890 heartbeater.cc:503] Master 127.7.157.124:34937 was elected leader, sending a full tablet report...
I20260929 18:42:44.428097 15270 tablet_bootstrap.cc:492] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d: No bootstrap required, opened a new log
I20260929 18:42:44.428196 15270 ts_tablet_manager.cc:1397] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 18:42:44.428684 15270 raft_consensus.cc:348] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } }
I20260929 18:42:44.428820 15270 raft_consensus.cc:374] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:44.428875 15270 raft_consensus.cc:729] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28610da940c495f9a268825f9994a3d, State: Initialized, Role: FOLLOWER
I20260929 18:42:44.429010 15270 consensus_queue.cc:260] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } }
I20260929 18:42:44.429389 15270 ts_tablet_manager.cc:1428] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:44.430037 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.429809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.67:60929)
0929 18:42:44.429915 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.429970 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 18:42:44.430617 15164 heartbeater.cc:503] Master 127.7.157.124:34937 was elected leader, sending a full tablet report...
I20260929 18:42:44.430756 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.430552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.67:55245)
0929 18:42:44.430639 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.430687 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 18:42:44.444978 15274 raft_consensus.cc:482] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:44.445125 15274 raft_consensus.cc:504] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } }
I20260929 18:42:44.445520 15274 leader_election.cc:288] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ec3bc489671458189f218079f60c6ff (127.7.157.66:42467), e28610da940c495f9a268825f9994a3d (127.7.157.67:37959)
I20260929 18:42:44.445928 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ea18bbb9e14da4b0f8d0e2acfe2392" candidate_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ec3bc489671458189f218079f60c6ff" is_pre_election: true
I20260929 18:42:44.445948 15118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ea18bbb9e14da4b0f8d0e2acfe2392" candidate_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28610da940c495f9a268825f9994a3d" is_pre_election: true
I20260929 18:42:44.446074 15118 raft_consensus.cc:2454] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a8d904fb2c54e72bbf064ec58d2a15a in term 0.
I20260929 18:42:44.446084 14981 raft_consensus.cc:2454] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a8d904fb2c54e72bbf064ec58d2a15a in term 0.
I20260929 18:42:44.446408 14778 leader_election.cc:302] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec3bc489671458189f218079f60c6ff, 5a8d904fb2c54e72bbf064ec58d2a15a; no voters:
I20260929 18:42:44.446609 15274 raft_consensus.cc:2781] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:44.446678 15274 raft_consensus.cc:482] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:44.446734 15274 raft_consensus.cc:3037] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:44.447808 15274 raft_consensus.cc:504] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } }
I20260929 18:42:44.448141 15274 leader_election.cc:288] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [CANDIDATE]: Term 1 election: Requested vote from peers 4ec3bc489671458189f218079f60c6ff (127.7.157.66:42467), e28610da940c495f9a268825f9994a3d (127.7.157.67:37959)
I20260929 18:42:44.448501 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ea18bbb9e14da4b0f8d0e2acfe2392" candidate_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ec3bc489671458189f218079f60c6ff"
I20260929 18:42:44.448508 15118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ea18bbb9e14da4b0f8d0e2acfe2392" candidate_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28610da940c495f9a268825f9994a3d"
I20260929 18:42:44.448617 15118 raft_consensus.cc:3037] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:44.448622 14981 raft_consensus.cc:3037] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:44.449491 14981 raft_consensus.cc:2454] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a8d904fb2c54e72bbf064ec58d2a15a in term 1.
I20260929 18:42:44.449519 15118 raft_consensus.cc:2454] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a8d904fb2c54e72bbf064ec58d2a15a in term 1.
I20260929 18:42:44.449762 14778 leader_election.cc:302] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec3bc489671458189f218079f60c6ff, 5a8d904fb2c54e72bbf064ec58d2a15a; no voters:
I20260929 18:42:44.449916 15274 raft_consensus.cc:2781] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:44.450122 15274 raft_consensus.cc:686] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [term 1 LEADER]: Becoming Leader. State: Replica: 5a8d904fb2c54e72bbf064ec58d2a15a, State: Running, Role: LEADER
I20260929 18:42:44.450261 15274 consensus_queue.cc:237] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } }
I20260929 18:42:44.450953 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.450693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.65:37459)
0929 18:42:44.450806 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.450869 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 18:42:44.452035 14707 catalog_manager.cc:5495] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a8d904fb2c54e72bbf064ec58d2a15a (127.7.157.65). New cstate: current_term: 1 leader_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a8d904fb2c54e72bbf064ec58d2a15a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 45901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20260929 18:42:44.469012 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.466309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.7.157.65:45901)
0929 18:42:44.466436 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:44.466440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:44.466454 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:42:44.466521 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:44.466524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:44.466578 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:44.466646 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:44.466851 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:44.467423 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:44.468203 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:44.468416 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:44.468481 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:44.468657 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:44.468723 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:44.468775 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:44.468855 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 18:42:44.468901 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 18:42:44.470170 15167 tablet.cc:2369] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:42:44.470928 15118 raft_consensus.cc:1261] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d [term 1 FOLLOWER]: Refusing update from remote peer 5a8d904fb2c54e72bbf064ec58d2a15a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:44.471110 14981 raft_consensus.cc:1261] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff [term 1 FOLLOWER]: Refusing update from remote peer 5a8d904fb2c54e72bbf064ec58d2a15a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:44.471385 15274 consensus_queue.cc:1035] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e28610da940c495f9a268825f9994a3d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:44.471524 15277 consensus_queue.cc:1035] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ec3bc489671458189f218079f60c6ff" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:44.472959 15027 heartbeater.cc:503] Master 127.7.157.124:34937 was elected leader, sending a full tablet report...
I20260929 18:42:44.473073 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.472794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.66:39285)
0929 18:42:44.472950 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.473005 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 18:42:44.473074 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.472811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.67:55063)
0929 18:42:44.472953 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.473001 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 18:42:44.476434 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.475832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.65:46801)
0929 18:42:44.476293 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.476351 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390}
I20260929 18:42:44.476944 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.476689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.67:36663)
0929 18:42:44.476828 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.476879 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 18:42:44.477972 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.477661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.66:54139)
0929 18:42:44.477853 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.477904 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260929 18:42:44.478749 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.478421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45147 (local address 127.7.157.65:44563)
0929 18:42:44.478608 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.478659 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260929 18:42:44.483331 14707 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:44.483480 14707 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:44.486853 14707 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa02076e43f842069ef4502c6855b606: Sending DeleteTablet for 3 replicas of tablet 00ea18bbb9e14da4b0f8d0e2acfe2392
I20260929 18:42:44.487553 14961 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ea18bbb9e14da4b0f8d0e2acfe2392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:44 UTC) from {username='oryx', principal='oryx/127.7.157.124@KRBTEST.COM'} at 127.0.0.1:41252
I20260929 18:42:44.487565 14824 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ea18bbb9e14da4b0f8d0e2acfe2392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:44 UTC) from {username='oryx', principal='oryx/127.7.157.124@KRBTEST.COM'} at 127.0.0.1:38596
I20260929 18:42:44.487690 15098 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ea18bbb9e14da4b0f8d0e2acfe2392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:44 UTC) from {username='oryx', principal='oryx/127.7.157.124@KRBTEST.COM'} at 127.0.0.1:46982
I20260929 18:42:44.487923 15290 tablet_replica.cc:330] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a: stopping tablet replica
I20260929 18:42:44.488051 15290 raft_consensus.cc:2229] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:42:44.488147 15292 tablet_replica.cc:330] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d: stopping tablet replica
I20260929 18:42:44.488277 15290 raft_consensus.cc:2258] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 5a8d904fb2c54e72bbf064ec58d2a15a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:44.488345 15292 raft_consensus.cc:2229] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:44.488476 15292 raft_consensus.cc:2258] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:44.488502 15291 tablet_replica.cc:330] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff: stopping tablet replica
I20260929 18:42:44.488618 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 14665
I20260929 18:42:44.488624 15291 raft_consensus.cc:2229] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:44.489656 15292 ts_tablet_manager.cc:1905] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:44.490909 15291 raft_consensus.cc:2258] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:44.491627 15291 ts_tablet_manager.cc:1905] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:44.492192 15292 ts_tablet_manager.cc:1918] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:44.492281 15292 log.cc:1196] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/00ea18bbb9e14da4b0f8d0e2acfe2392
I20260929 18:42:44.492561 15292 ts_tablet_manager.cc:1939] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P e28610da940c495f9a268825f9994a3d: Deleting consensus metadata
I20260929 18:42:44.493310 14697 catalog_manager.cc:4841] TS e28610da940c495f9a268825f9994a3d (127.7.157.67:37959): tablet 00ea18bbb9e14da4b0f8d0e2acfe2392 (table test-table [id=8f1b1226812b4c0d94d8548b44450dab]) successfully deleted
I20260929 18:42:44.493935 15291 ts_tablet_manager.cc:1918] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:44.494000 15291 log.cc:1196] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/00ea18bbb9e14da4b0f8d0e2acfe2392
I20260929 18:42:44.494202 15291 ts_tablet_manager.cc:1939] T 00ea18bbb9e14da4b0f8d0e2acfe2392 P 4ec3bc489671458189f218079f60c6ff: Deleting consensus metadata
I20260929 18:42:44.495204 14696 catalog_manager.cc:4841] TS 4ec3bc489671458189f218079f60c6ff (127.7.157.66:42467): tablet 00ea18bbb9e14da4b0f8d0e2acfe2392 (table test-table [id=8f1b1226812b4c0d94d8548b44450dab]) successfully deleted
I20260929 18:42:44.495584 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 14898
I20260929 18:42:44.495707 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:44.495482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:45901 (local address 127.0.0.1:38620)
0929 18:42:44.495582 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:44.495627 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 18:42:44.496394 14697 catalog_manager.cc:4584] TS 5a8d904fb2c54e72bbf064ec58d2a15a (127.7.157.65:45901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00ea18bbb9e14da4b0f8d0e2acfe2392: Network error: Client connection negotiation failed: client connection to 127.7.157.65:45901: connect: Connection refused (error 111)
I20260929 18:42:44.501094 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 15034
I20260929 18:42:44.506138 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 14383
I20260929 18:42:44.510798 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 14486
2026-09-29T18:42:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6725 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:42:44 dist-test-slave-smpn krb5kdc[15295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:42:44 dist-test-slave-smpn krb5kdc[15295](info): set up 2 sockets
Sep 29 18:42:44 dist-test-slave-smpn krb5kdc[15295](info): commencing operation
krb5kdc: starting...
W20260929 18:42:45.471385 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:42:45 dist-test-slave-smpn krb5kdc[15295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:42:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.126.keytab.
Entry for principal oryx/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.126.keytab.
I20260929 18:42:45.590375 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:37083
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.126.keytab
--principal=oryx/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:37083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:45.697193 15308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:45.697474 15308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:45.697530 15308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:45.701538 15308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:45.701596 15308 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:42:45.701617 15308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:45.701642 15308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:45.701663 15308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:45.701689 15308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:45.707224 15308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:37083
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.126.keytab
--principal=oryx/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:37083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:45.708285 15308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:45.709285 15308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:45.717706 15314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:45.717706 15316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:45.717799 15308 server_base.cc:1034] running on GCE node
W20260929 18:42:45.717710 15313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:45.718601 15308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:45.719717 15308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:45.720871 15308 hybrid_clock.cc:648] HybridClock initialized: now 1790707365720855 us; error 28 us; skew 500 ppm
Sep 29 18:42:45 dist-test-slave-smpn krb5kdc[15295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707365, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:45.723562 15308 init.cc:377] Logged in from keytab as oryx/127.7.157.126@KRBTEST.COM (short username oryx)
I20260929 18:42:45.724431 15308 webserver.cc:460] Webserver started at http://127.7.157.126:42033/ using document root <none> and password file <none>
I20260929 18:42:45.724951 15308 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:45.725010 15308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:45.725203 15308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:45.727340 15308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "4f16ebedc51f46a69c11a45812442233"
format_stamp: "Formatted at 2026-09-29 18:42:45 on dist-test-slave-smpn"
server_key: "774049055527e294a513570851388425"
server_key_iv: "032f7fa5770fc28972ddda9566eb0061"
server_key_version: "encryptionkey@0"
I20260929 18:42:45.727864 15308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "4f16ebedc51f46a69c11a45812442233"
format_stamp: "Formatted at 2026-09-29 18:42:45 on dist-test-slave-smpn"
server_key: "774049055527e294a513570851388425"
server_key_iv: "032f7fa5770fc28972ddda9566eb0061"
server_key_version: "encryptionkey@0"
I20260929 18:42:45.731441 15308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:42:45.733724 15323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:45.734927 15308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:45.735039 15308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "4f16ebedc51f46a69c11a45812442233"
format_stamp: "Formatted at 2026-09-29 18:42:45 on dist-test-slave-smpn"
server_key: "774049055527e294a513570851388425"
server_key_iv: "032f7fa5770fc28972ddda9566eb0061"
server_key_version: "encryptionkey@0"
I20260929 18:42:45.735175 15308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:45.747891 15308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:45.750730 15308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:45.750900 15308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:45.758572 15308 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:37083
I20260929 18:42:45.758595 15385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:37083 every 8 connection(s)
I20260929 18:42:45.759559 15308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:45.762440 15386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:45.766505 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 15308
I20260929 18:42:45.766624 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:45.766888 7797 external_mini_cluster.cc:1428] Setting key 5d6a632f7f0dc8be8f397d227b12ae0f
I20260929 18:42:45.768250 15386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233: Bootstrap starting.
I20260929 18:42:45.770614 15386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:45.771416 15386 log.cc:824] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:45.773206 15386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233: No bootstrap required, opened a new log
Sep 29 18:42:45 dist-test-slave-smpn krb5kdc[15295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM
I20260929 18:42:45.776791 15386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f16ebedc51f46a69c11a45812442233" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37083 } }
I20260929 18:42:45.777024 15386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:45.777079 15386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f16ebedc51f46a69c11a45812442233, State: Initialized, Role: FOLLOWER
I20260929 18:42:45.777529 15386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f16ebedc51f46a69c11a45812442233" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37083 } }
I20260929 18:42:45.777693 15386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:45.777760 15386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:45.777853 15386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:45.779007 15386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f16ebedc51f46a69c11a45812442233" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37083 } }
I20260929 18:42:45.779378 15386 leader_election.cc:302] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f16ebedc51f46a69c11a45812442233; no voters:
I20260929 18:42:45.779738 15386 leader_election.cc:288] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:45.779830 15391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:45.780081 15391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [term 1 LEADER]: Becoming Leader. State: Replica: 4f16ebedc51f46a69c11a45812442233, State: Running, Role: LEADER
I20260929 18:42:45.780436 15391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f16ebedc51f46a69c11a45812442233" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37083 } }
I20260929 18:42:45.780562 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:45.768223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.7.157.126:37083)
0929 18:42:45.768639 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:45.768655 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:45.768692 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:42:45.769340 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:45.769357 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:45.769696 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:45.769973 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:45.770855 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:45.771912 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:45.772738 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:45.773201 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:45.773358 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:45.775747 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:45.775753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:45.775768 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:45.775796 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:45.777388 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:45.777874 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:45.777881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:45.777886 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:45.778033 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:45.778272 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:45.778275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:45.778277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:45.778790 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:45.778991 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:45.779220 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 18:42:45.779522 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":125,"threads_started":1}
I20260929 18:42:45.781005 15386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:45.782027 15392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f16ebedc51f46a69c11a45812442233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f16ebedc51f46a69c11a45812442233" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37083 } } }
I20260929 18:42:45.782143 15392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:45.782542 15393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f16ebedc51f46a69c11a45812442233. Latest consensus state: current_term: 1 leader_uuid: "4f16ebedc51f46a69c11a45812442233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f16ebedc51f46a69c11a45812442233" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37083 } } }
I20260929 18:42:45.782771 15393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233 [sys.catalog]: This master's current role is: LEADER
W20260929 18:42:45.785686 15406 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:42:45.785749 15406 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:42:45.785818 15400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:45.786543 15400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:45.792140 15400 catalog_manager.cc:1347] Generated new cluster ID: 01cc43c2b8b04e92a0412e94147ca7de
I20260929 18:42:45.792217 15400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:45.823966 15400 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:45.825253 15400 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:45.831385 15400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233: Generated new TSK 0
I20260929 18:42:45.832207 15400 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.65.keytab.
Entry for principal oryx/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.65.keytab.
I20260929 18:42:45.858245 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:37083
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.65.keytab
--principal=oryx/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:45.968163 15412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:45.968441 15412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:45.968499 15412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:45.972456 15412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:45.972517 15412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:45.972568 15412 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:42:45.972615 15412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:42:45.978102 15412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.65.keytab
--principal=oryx/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.157.126:37083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:45.979218 15412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:45.980242 15412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:42:45.989719 15412 server_base.cc:1034] running on GCE node
W20260929 18:42:45.989689 15420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:45.989687 15417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:45.989687 15418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:45.990409 15412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:45.991031 15412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:45.992206 15412 hybrid_clock.cc:648] HybridClock initialized: now 1790707365992189 us; error 31 us; skew 500 ppm
Sep 29 18:42:45 dist-test-slave-smpn krb5kdc[15295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707365, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:45.994860 15412 init.cc:377] Logged in from keytab as oryx/127.7.157.65@KRBTEST.COM (short username oryx)
I20260929 18:42:45.995745 15412 webserver.cc:460] Webserver started at http://127.7.157.65:41499/ using document root <none> and password file <none>
I20260929 18:42:45.996256 15412 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:45.996311 15412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:45.996469 15412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:45.998466 15412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "f758aaacf28f4cb288225ed5af1dcfbd"
format_stamp: "Formatted at 2026-09-29 18:42:45 on dist-test-slave-smpn"
server_key: "eef970cc622ec55bd5e1a5691f2415cf"
server_key_iv: "d6a9af1ac5b2c8119026202043990e62"
server_key_version: "encryptionkey@0"
I20260929 18:42:45.998979 15412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "f758aaacf28f4cb288225ed5af1dcfbd"
format_stamp: "Formatted at 2026-09-29 18:42:45 on dist-test-slave-smpn"
server_key: "eef970cc622ec55bd5e1a5691f2415cf"
server_key_iv: "d6a9af1ac5b2c8119026202043990e62"
server_key_version: "encryptionkey@0"
I20260929 18:42:46.002569 15412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260929 18:42:46.004926 15427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:46.006114 15412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:46.006213 15412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "f758aaacf28f4cb288225ed5af1dcfbd"
format_stamp: "Formatted at 2026-09-29 18:42:45 on dist-test-slave-smpn"
server_key: "eef970cc622ec55bd5e1a5691f2415cf"
server_key_iv: "d6a9af1ac5b2c8119026202043990e62"
server_key_version: "encryptionkey@0"
I20260929 18:42:46.006320 15412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:46.016789 15412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:46.019745 15412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:46.019914 15412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:46.021306 15412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:46.021361 15412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:46.021400 15412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:46.021423 15412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:42:46 dist-test-slave-smpn krb5kdc[15295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707365, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM
I20260929 18:42:46.031730 15412 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:37289
I20260929 18:42:46.031782 15542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:37289 every 8 connection(s)
I20260929 18:42:46.032747 15412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:42:46.034480 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 15412
I20260929 18:42:46.034610 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:42:46.034909 7797 external_mini_cluster.cc:1428] Setting key c4d35ae64804ef71ffcb8f43350e3fe5
I20260929 18:42:46.036064 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.022186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:35381 (local address 127.7.157.126:37083)
0929 18:42:46.022358 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:46.022363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:46.023331 (+ 968us) server_negotiation.cc:407] Connection header received
0929 18:42:46.024599 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:46.024603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:46.024667 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:46.024770 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:46.026253 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.027226 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.028761 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.029105 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.029200 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.032365 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:46.032368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:46.032370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:46.032395 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:46.033934 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:46.034541 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:46.034544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:46.034546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:46.034642 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:46.035005 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:46.035008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:46.035010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:46.035227 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:46.035316 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:46.035815 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 18:42:46.035928 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 18:42:46.036989 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.022642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:37083 (local address 127.7.157.65:35381)
0929 18:42:46.023156 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:46.023211 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:46.024301 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:46.024991 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:46.024998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:46.025424 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:46.026088 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.026106 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.027361 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.027364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.028600 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.028612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.029417 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.029421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.029622 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.030328 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:46.030351 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:46.032190 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:46.034110 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:46.034115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:46.034128 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:46.034441 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:46.034737 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:46.034740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:46.034742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:46.034916 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:46.035328 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:46.035335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:46.035582 (+ 247us) client_negotiation.cc:769] Sending connection context
0929 18:42:46.035783 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 18:42:46.036106 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":113,"threads_started":1}
I20260929 18:42:46.038170 15543 heartbeater.cc:347] Connected to a master server at 127.7.157.126:37083
I20260929 18:42:46.038306 15543 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:46.038877 15543 heartbeater.cc:511] Master 127.7.157.126:37083 requested a full tablet report, sending...
I20260929 18:42:46.040396 15339 ts_manager.cc:194] Registered new tserver with Master: f758aaacf28f4cb288225ed5af1dcfbd (127.7.157.65:37289)
I20260929 18:42:46.041762 15339 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:35381
WARNING: no policy specified for oryx/127.7.157.66@KRBTEST.COM; defaulting to no policy
I20260929 18:42:46.047796 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.040426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:55089 (local address 127.7.157.126:37083)
0929 18:42:46.040536 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:46.040541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:46.040736 (+ 195us) server_negotiation.cc:407] Connection header received
0929 18:42:46.040828 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:46.040831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:46.040870 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:46.040949 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:46.041430 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.042129 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.043310 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.043622 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.043725 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.044912 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:46.044915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:46.044917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:46.044937 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:46.046226 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:46.046670 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:46.046674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:46.046676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:46.046757 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:46.047008 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:46.047009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:46.047011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:46.047212 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:46.047288 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:46.047543 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 18:42:46.047650 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:42:46.047803 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.040343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:37083 (local address 127.7.157.65:55089)
0929 18:42:46.040649 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:46.040663 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:46.040759 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:46.040973 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:46.040976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:46.041201 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:46.041322 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.041329 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.042276 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.042280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.043192 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.043204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.043751 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.043755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.043860 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.044310 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:46.044324 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:46.044771 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:46.046332 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:46.046336 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:46.046339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:46.046580 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:46.046837 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:46.046840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:46.046842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:46.046933 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:46.047317 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:46.047320 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:46.047402 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 18:42:46.047494 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 18:42:46.047644 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.66.keytab.
Entry for principal oryx/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.66.keytab.
I20260929 18:42:46.065817 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:37083
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.66.keytab
--principal=oryx/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:46.172217 15552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:46.172469 15552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:46.172513 15552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:46.176393 15552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:46.176440 15552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:46.176476 15552 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:42:46.176512 15552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:42:46.181594 15552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.66.keytab
--principal=oryx/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.157.126:37083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:46.182564 15552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:46.183555 15552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:46.192674 15557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:46.192668 15560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:46.192770 15552 server_base.cc:1034] running on GCE node
W20260929 18:42:46.192695 15558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:46.193259 15552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:46.193867 15552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:46.195011 15552 hybrid_clock.cc:648] HybridClock initialized: now 1790707366194983 us; error 45 us; skew 500 ppm
Sep 29 18:42:46 dist-test-slave-smpn krb5kdc[15295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707366, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:46.197707 15552 init.cc:377] Logged in from keytab as oryx/127.7.157.66@KRBTEST.COM (short username oryx)
I20260929 18:42:46.198647 15552 webserver.cc:460] Webserver started at http://127.7.157.66:41149/ using document root <none> and password file <none>
I20260929 18:42:46.199229 15552 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:46.199294 15552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:46.199493 15552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:46.201576 15552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "da3767d8ac684ce193d63744cf6ffaf9"
format_stamp: "Formatted at 2026-09-29 18:42:46 on dist-test-slave-smpn"
server_key: "9ec69e29129660ff196cf7fde8a0e965"
server_key_iv: "4e08164bef3f56944729d1b04f8bc790"
server_key_version: "encryptionkey@0"
I20260929 18:42:46.202107 15552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "da3767d8ac684ce193d63744cf6ffaf9"
format_stamp: "Formatted at 2026-09-29 18:42:46 on dist-test-slave-smpn"
server_key: "9ec69e29129660ff196cf7fde8a0e965"
server_key_iv: "4e08164bef3f56944729d1b04f8bc790"
server_key_version: "encryptionkey@0"
I20260929 18:42:46.205875 15552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:42:46.208444 15567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:46.209736 15552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:46.209899 15552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "da3767d8ac684ce193d63744cf6ffaf9"
format_stamp: "Formatted at 2026-09-29 18:42:46 on dist-test-slave-smpn"
server_key: "9ec69e29129660ff196cf7fde8a0e965"
server_key_iv: "4e08164bef3f56944729d1b04f8bc790"
server_key_version: "encryptionkey@0"
I20260929 18:42:46.210028 15552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:46.225052 15552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:46.228266 15552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:46.228459 15552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:46.229935 15552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:46.229987 15552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:46.230041 15552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:46.230068 15552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:42:46 dist-test-slave-smpn krb5kdc[15295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707366, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM
I20260929 18:42:46.240028 15552 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:45709
I20260929 18:42:46.240366 15682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:45709 every 8 connection(s)
I20260929 18:42:46.240978 15552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:42:46.241295 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 15552
I20260929 18:42:46.241417 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:42:46.241722 7797 external_mini_cluster.cc:1428] Setting key b4ecb40338bc4ad53346ddd7c28ac34f
I20260929 18:42:46.243245 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.230690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:59987 (local address 127.7.157.126:37083)
0929 18:42:46.230828 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:46.230833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:46.231784 (+ 951us) server_negotiation.cc:407] Connection header received
0929 18:42:46.232864 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:46.232867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:46.232925 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:46.232997 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:46.234402 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.235037 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.236329 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.236561 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.236650 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.239509 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:46.239513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:46.239515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:46.239542 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:46.240957 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:46.241542 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:46.241545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:46.241547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:46.241653 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:46.242022 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:46.242025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:46.242027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:46.242296 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:46.242383 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:46.242929 (+ 546us) server_negotiation.cc:299] Negotiation successful
0929 18:42:46.243050 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:42:46.244263 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.231171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:37083 (local address 127.7.157.66:59987)
0929 18:42:46.231620 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:46.231670 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:46.232626 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:46.233312 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:46.233322 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:46.233736 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:46.234258 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.234272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.235217 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.235221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.236216 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.236226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.236712 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.236716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.236880 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.237435 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:46.237451 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:46.239340 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:46.241067 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:46.241072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:46.241087 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:46.241428 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:46.241756 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:46.241759 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:46.241761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:46.241916 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:46.242400 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:46.242406 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:46.242704 (+ 298us) client_negotiation.cc:769] Sending connection context
0929 18:42:46.242905 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 18:42:46.243268 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":112,"threads_started":1}
I20260929 18:42:46.245620 15683 heartbeater.cc:347] Connected to a master server at 127.7.157.126:37083
I20260929 18:42:46.245779 15683 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:46.246327 15683 heartbeater.cc:511] Master 127.7.157.126:37083 requested a full tablet report, sending...
I20260929 18:42:46.247470 15340 ts_manager.cc:194] Registered new tserver with Master: da3767d8ac684ce193d63744cf6ffaf9 (127.7.157.66:45709)
I20260929 18:42:46.248184 15340 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:59987
WARNING: no policy specified for oryx/127.7.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.157.67@KRBTEST.COM" created.
I20260929 18:42:46.254552 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.247618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:54543 (local address 127.7.157.126:37083)
0929 18:42:46.247736 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:46.247741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:46.248093 (+ 352us) server_negotiation.cc:407] Connection header received
0929 18:42:46.248190 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:46.248193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:46.248236 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:46.248308 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:46.248922 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.249429 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.250214 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.250420 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.250501 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.251587 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:46.251591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:46.251593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:46.251617 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:46.252878 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:46.253461 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:46.253463 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:46.253465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:46.253537 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:46.253741 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:46.253744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:46.253745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:46.253948 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:46.254041 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:46.254283 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 18:42:46.254399 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:42:46.254559 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.247730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:37083 (local address 127.7.157.66:54543)
0929 18:42:46.247999 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:46.248014 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:46.248114 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:46.248510 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:46.248513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:46.248716 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:46.248816 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.248821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.249549 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.249552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.250113 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.250120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.250512 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.250515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.250599 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.251115 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:46.251129 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:46.251467 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:46.253149 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:46.253152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:46.253154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:46.253370 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:46.253608 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:46.253609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:46.253611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:46.253679 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:46.254033 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:46.254037 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:46.254125 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 18:42:46.254223 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 18:42:46.254366 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.67.keytab.
Entry for principal oryx/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.67.keytab.
I20260929 18:42:46.268662 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:37083
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.67.keytab
--principal=oryx/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:46.382885 15692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:46.383194 15692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:46.383255 15692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:46.387359 15692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:46.387420 15692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:46.387475 15692 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:42:46.387530 15692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:42:46.392844 15692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.67.keytab
--principal=oryx/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.157.126:37083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:46.393956 15692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:46.394961 15692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:46.404500 15698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:46.404500 15700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:46.404567 15697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:46.404578 15692 server_base.cc:1034] running on GCE node
I20260929 18:42:46.405072 15692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:46.405689 15692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:46.406842 15692 hybrid_clock.cc:648] HybridClock initialized: now 1790707366406825 us; error 32 us; skew 500 ppm
Sep 29 18:42:46 dist-test-slave-smpn krb5kdc[15295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707366, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:46.409564 15692 init.cc:377] Logged in from keytab as oryx/127.7.157.67@KRBTEST.COM (short username oryx)
I20260929 18:42:46.410418 15692 webserver.cc:460] Webserver started at http://127.7.157.67:39581/ using document root <none> and password file <none>
I20260929 18:42:46.410969 15692 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:46.411036 15692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:46.411267 15692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:46.413395 15692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "7348b1e81a0b44ecb05cdc321c763613"
format_stamp: "Formatted at 2026-09-29 18:42:46 on dist-test-slave-smpn"
server_key: "08c9d829eeae457f8c8fff97ac2a3417"
server_key_iv: "7a0ff1e2dc66bea3893d725883320986"
server_key_version: "encryptionkey@0"
I20260929 18:42:46.413952 15692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "7348b1e81a0b44ecb05cdc321c763613"
format_stamp: "Formatted at 2026-09-29 18:42:46 on dist-test-slave-smpn"
server_key: "08c9d829eeae457f8c8fff97ac2a3417"
server_key_iv: "7a0ff1e2dc66bea3893d725883320986"
server_key_version: "encryptionkey@0"
I20260929 18:42:46.417687 15692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:42:46.420162 15707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:46.421499 15692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:46.421608 15692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "7348b1e81a0b44ecb05cdc321c763613"
format_stamp: "Formatted at 2026-09-29 18:42:46 on dist-test-slave-smpn"
server_key: "08c9d829eeae457f8c8fff97ac2a3417"
server_key_iv: "7a0ff1e2dc66bea3893d725883320986"
server_key_version: "encryptionkey@0"
I20260929 18:42:46.421706 15692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:46.443396 15692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:46.446336 15692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:46.446503 15692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:46.447978 15692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:46.448032 15692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:46.448083 15692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:46.448110 15692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:42:46 dist-test-slave-smpn krb5kdc[15295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707366, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM
I20260929 18:42:46.457755 15692 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:43983
I20260929 18:42:46.457825 15822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:43983 every 8 connection(s)
I20260929 18:42:46.458824 15692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:42:46.460644 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.448711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:37415 (local address 127.7.157.126:37083)
0929 18:42:46.448859 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:46.448864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:46.449664 (+ 800us) server_negotiation.cc:407] Connection header received
0929 18:42:46.450803 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:46.450808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:46.450871 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:46.450972 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:46.452188 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.452721 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.453895 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.454170 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.454264 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.457055 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:46.457059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:46.457062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:46.457091 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:46.458619 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:46.459175 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:46.459179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:46.459181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:46.459284 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:46.459684 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:46.459687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:46.459689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:46.459878 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:46.459955 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:46.460411 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 18:42:46.460518 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:42:46.461376 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.449086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:37083 (local address 127.7.157.67:37415)
0929 18:42:46.449512 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:46.449559 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:46.450514 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:46.451171 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:46.451178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:46.451555 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:46.452047 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.452062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.452834 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.452837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.453773 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.453784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.454328 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.454332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.454523 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.455108 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:46.455127 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:46.456881 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:46.458745 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:46.458749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:46.458762 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:46.459055 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:46.459393 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:46.459396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:46.459398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:46.459563 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:46.459967 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:46.459972 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:46.460194 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 18:42:46.460345 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 18:42:46.460596 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260929 18:42:46.462594 15823 heartbeater.cc:347] Connected to a master server at 127.7.157.126:37083
I20260929 18:42:46.462720 15823 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:46.463214 15823 heartbeater.cc:511] Master 127.7.157.126:37083 requested a full tablet report, sending...
I20260929 18:42:46.464156 15339 ts_manager.cc:194] Registered new tserver with Master: 7348b1e81a0b44ecb05cdc321c763613 (127.7.157.67:43983)
I20260929 18:42:46.464823 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 15692
I20260929 18:42:46.464823 15339 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:37415
I20260929 18:42:46.464922 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:42:46.465221 7797 external_mini_cluster.cc:1428] Setting key 22e3f203c4846f55a6a5d5bd86001e3d
I20260929 18:42:46.467301 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:42:46.472268 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.464604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:54221 (local address 127.7.157.126:37083)
0929 18:42:46.465546 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:46.465551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:46.465863 (+ 312us) server_negotiation.cc:407] Connection header received
0929 18:42:46.466027 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:46.466040 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:46.466081 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:46.466145 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:46.466763 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.467339 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.468120 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.468328 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.468409 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.469570 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:46.469573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:46.469575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:46.469596 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:46.470829 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:46.471231 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:46.471234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:46.471236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:46.471307 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:46.471536 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:46.471539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:46.471541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:46.471702 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:46.471793 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:46.472020 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 18:42:46.472119 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868}
I20260929 18:42:46.472296 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.464507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:37083 (local address 127.7.157.67:54221)
0929 18:42:46.465695 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:46.465709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:46.465935 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:46.466365 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:46.466367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:46.466541 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:46.466655 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.466662 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.467473 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.467476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.468018 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.468025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.468420 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.468423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.468508 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.468993 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:46.469008 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:46.469408 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:46.470924 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:46.470927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:46.470929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:46.471153 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:46.471382 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:46.471385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:46.471387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:46.471464 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:46.471793 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:46.471796 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:46.471881 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 18:42:46.471964 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:42:46.472097 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"thread_start_us":85,"threads_started":1}
Sep 29 18:42:46 dist-test-slave-smpn krb5kdc[15295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707366, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:42:46 dist-test-slave-smpn krb5kdc[15295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:46 dist-test-slave-smpn krb5kdc[15295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:42:46.490202 15389 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:46.484458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.7.157.126:37083)
0929 18:42:46.484584 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:46.484588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:46.484599 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:42:46.484638 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:46.484640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:46.484678 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:46.484753 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:46.485435 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.485952 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.486766 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.486976 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.487060 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.490073 (+ 3013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32892: BlockingRecv error: recv got EOF from 127.0.0.1:32892 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 18:42:46 dist-test-slave-smpn krb5kdc[15295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707366, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM
I20260929 18:42:46.498713 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.491522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.7.157.126:37083)
0929 18:42:46.491643 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:46.491647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:46.491752 (+ 105us) server_negotiation.cc:407] Connection header received
0929 18:42:46.491869 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:46.491872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:46.491913 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:46.491972 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:46.492519 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.493016 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.493763 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.493976 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.494060 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.496184 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:46.496187 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:46.496190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:46.496215 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:46.497289 (+ 1074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:46.497643 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:46.497646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:46.497648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:46.497721 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:46.497964 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:46.497967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:46.497969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:46.498135 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:46.498246 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:46.498466 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 18:42:46.498566 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:42:46.501322 15339 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:42:46.504243 15339 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:46.517155 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.511913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60438 (local address 127.7.157.67:43983)
0929 18:42:46.512579 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:46.512591 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:46.512616 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:42:46.512691 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:46.512696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:46.512913 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:46.513100 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:46.513323 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.514388 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.515720 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.516513 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.516612 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.516799 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:46.516910 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 18:42:46.517002 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":104,"threads_started":1}
I20260929 18:42:46.517158 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.511813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:43983 (local address 127.0.0.1:60438)
0929 18:42:46.512139 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:46.512191 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:46.512315 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:46.513081 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:46.513087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:46.513112 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:46.513224 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.513228 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.514513 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.514516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.515612 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.515624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.516624 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.516627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.516736 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.516774 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 18:42:46.516886 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 18:42:46.517009 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":70,"threads_started":1}
I20260929 18:42:46.517892 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.512163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:37289 (local address 127.0.0.1:51478)
0929 18:42:46.512757 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:46.512769 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:46.512851 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:46.513840 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:46.513843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:46.513861 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:46.513956 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.513962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.515297 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.515301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.516472 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.516483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.517506 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.517510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.517600 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.517622 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:42:46.517698 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:42:46.517765 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260929 18:42:46.517963 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.512875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.7.157.65:37289)
0929 18:42:46.513446 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:46.513461 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:46.513481 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:42:46.513549 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:46.513554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:46.513708 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:46.513854 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:46.514088 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.515163 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.516625 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.517401 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.517494 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.517640 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:46.517744 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 18:42:46.517834 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":117,"threads_started":1}
I20260929 18:42:46.518347 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.512503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:45709 (local address 127.0.0.1:47874)
0929 18:42:46.513465 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:46.513478 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:46.513576 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:46.514198 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:46.514201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:46.514268 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:46.514348 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.514354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.515867 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.515870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.516823 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.516830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.518007 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.518011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.518096 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.518117 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:42:46.518193 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:42:46.518259 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"thread_start_us":89,"threads_started":1}
I20260929 18:42:46.519033 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.512684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47874 (local address 127.7.157.66:45709)
0929 18:42:46.513089 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:46.513100 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:46.513667 (+ 567us) server_negotiation.cc:407] Connection header received
0929 18:42:46.513740 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:46.513745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:46.513928 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:46.514061 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:46.514459 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.515549 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.516942 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.517719 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.517804 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.517946 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:46.518812 (+ 866us) server_negotiation.cc:299] Negotiation successful
0929 18:42:46.518904 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":106,"threads_started":1}
I20260929 18:42:46.520470 15757 tablet_service.cc:1467] Processing CreateTablet for tablet 3f63baea9982483fb6d2489259c5a58e (DEFAULT_TABLE table=test-table [id=8fb4ec32bfc049caa6636eb1e94477d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:46.520720 15477 tablet_service.cc:1467] Processing CreateTablet for tablet 3f63baea9982483fb6d2489259c5a58e (DEFAULT_TABLE table=test-table [id=8fb4ec32bfc049caa6636eb1e94477d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:46.521430 15757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f63baea9982483fb6d2489259c5a58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:46.521703 15477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f63baea9982483fb6d2489259c5a58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:46.521694 15617 tablet_service.cc:1467] Processing CreateTablet for tablet 3f63baea9982483fb6d2489259c5a58e (DEFAULT_TABLE table=test-table [id=8fb4ec32bfc049caa6636eb1e94477d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:46.522595 15617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f63baea9982483fb6d2489259c5a58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:46.527916 15851 tablet_bootstrap.cc:492] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613: Bootstrap starting.
I20260929 18:42:46.529811 15853 tablet_bootstrap.cc:492] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9: Bootstrap starting.
I20260929 18:42:46.530071 15851 tablet_bootstrap.cc:654] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:46.530984 15851 log.cc:824] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:46.531399 15852 tablet_bootstrap.cc:492] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd: Bootstrap starting.
I20260929 18:42:46.531908 15853 tablet_bootstrap.cc:654] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:46.532543 15853 log.cc:824] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:46.533084 15851 tablet_bootstrap.cc:492] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613: No bootstrap required, opened a new log
I20260929 18:42:46.533272 15851 ts_tablet_manager.cc:1397] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260929 18:42:46.533347 15852 tablet_bootstrap.cc:654] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:46.534113 15852 log.cc:824] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:46.534154 15853 tablet_bootstrap.cc:492] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9: No bootstrap required, opened a new log
I20260929 18:42:46.534327 15853 ts_tablet_manager.cc:1397] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:42:46.536020 15852 tablet_bootstrap.cc:492] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd: No bootstrap required, opened a new log
I20260929 18:42:46.536206 15852 ts_tablet_manager.cc:1397] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:42:46.536787 15851 raft_consensus.cc:348] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } }
I20260929 18:42:46.537004 15851 raft_consensus.cc:374] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:46.537074 15851 raft_consensus.cc:729] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7348b1e81a0b44ecb05cdc321c763613, State: Initialized, Role: FOLLOWER
I20260929 18:42:46.537497 15851 consensus_queue.cc:260] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } }
I20260929 18:42:46.538198 15823 heartbeater.cc:503] Master 127.7.157.126:37083 was elected leader, sending a full tablet report...
I20260929 18:42:46.538120 15853 raft_consensus.cc:348] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } }
I20260929 18:42:46.538386 15853 raft_consensus.cc:374] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:46.538439 15853 raft_consensus.cc:729] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3767d8ac684ce193d63744cf6ffaf9, State: Initialized, Role: FOLLOWER
I20260929 18:42:46.538452 15851 ts_tablet_manager.cc:1428] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613: Time spent starting tablet: real 0.005s user 0.004s sys 0.003s
I20260929 18:42:46.538888 15853 consensus_queue.cc:260] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } }
I20260929 18:42:46.539613 15683 heartbeater.cc:503] Master 127.7.157.126:37083 was elected leader, sending a full tablet report...
I20260929 18:42:46.539940 15853 ts_tablet_manager.cc:1428] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:42:46.540609 15852 raft_consensus.cc:348] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } }
I20260929 18:42:46.540843 15852 raft_consensus.cc:374] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:46.540936 15852 raft_consensus.cc:729] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f758aaacf28f4cb288225ed5af1dcfbd, State: Initialized, Role: FOLLOWER
I20260929 18:42:46.541363 15852 consensus_queue.cc:260] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } }
I20260929 18:42:46.542315 15852 ts_tablet_manager.cc:1428] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:42:46.542642 15543 heartbeater.cc:503] Master 127.7.157.126:37083 was elected leader, sending a full tablet report...
W20260929 18:42:46.709815 15824 tablet.cc:2369] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:42:46.742378 15684 tablet.cc:2369] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:42:46.784322 15544 tablet.cc:2369] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:42:46.976918 15858 raft_consensus.cc:482] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:46.977123 15858 raft_consensus.cc:504] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } }
I20260929 18:42:46.978192 15858 leader_election.cc:288] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7348b1e81a0b44ecb05cdc321c763613 (127.7.157.67:43983), f758aaacf28f4cb288225ed5af1dcfbd (127.7.157.65:37289)
I20260929 18:42:46.982059 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.978459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:37289 (local address 127.7.157.66:36905)
0929 18:42:46.978772 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:46.978783 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:46.978883 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:46.979133 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:46.979135 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:46.979175 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:46.979294 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.979299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.980046 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.980062 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.981001 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.981009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.981675 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.981678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.981772 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.981792 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:42:46.981850 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:42:46.981901 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":61,"spinlock_wait_cycles":2048,"thread_start_us":60,"threads_started":1}
I20260929 18:42:46.982075 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.978560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:36905 (local address 127.7.157.65:37289)
0929 18:42:46.978681 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:46.978685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:46.978862 (+ 177us) server_negotiation.cc:407] Connection header received
0929 18:42:46.978976 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:46.978978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:46.979017 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:46.979120 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:46.979396 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.979941 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.981122 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.981560 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.981672 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.981713 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:46.981884 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 18:42:46.981952 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:42:46.982172 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.978511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:56615 (local address 127.7.157.67:43983)
0929 18:42:46.978683 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:46.978687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:46.978816 (+ 129us) server_negotiation.cc:407] Connection header received
0929 18:42:46.978942 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:46.978945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:46.978991 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:46.979044 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:46.979356 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.979916 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.981100 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.981525 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.981663 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.981714 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:46.982000 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 18:42:46.982067 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 18:42:46.982213 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:46.978417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:43983 (local address 127.7.157.66:56615)
0929 18:42:46.978712 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:46.978726 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:46.978843 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:46.979095 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:46.979097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:46.979114 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:46.979266 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.979271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.980012 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.980015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.980993 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:46.981002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:46.981795 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:46.981798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:46.981884 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:46.981907 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:42:46.981974 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:42:46.982039 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":46,"spinlock_wait_cycles":1408,"thread_start_us":80,"threads_started":1}
I20260929 18:42:46.982815 15497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f63baea9982483fb6d2489259c5a58e" candidate_uuid: "da3767d8ac684ce193d63744cf6ffaf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" is_pre_election: true
I20260929 18:42:46.982889 15777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f63baea9982483fb6d2489259c5a58e" candidate_uuid: "da3767d8ac684ce193d63744cf6ffaf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7348b1e81a0b44ecb05cdc321c763613" is_pre_election: true
I20260929 18:42:46.983136 15497 raft_consensus.cc:2454] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3767d8ac684ce193d63744cf6ffaf9 in term 0.
I20260929 18:42:46.983160 15777 raft_consensus.cc:2454] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3767d8ac684ce193d63744cf6ffaf9 in term 0.
I20260929 18:42:46.983560 15570 leader_election.cc:302] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3767d8ac684ce193d63744cf6ffaf9, f758aaacf28f4cb288225ed5af1dcfbd; no voters:
I20260929 18:42:46.983810 15858 raft_consensus.cc:2781] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:46.983888 15858 raft_consensus.cc:482] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:46.983930 15858 raft_consensus.cc:3037] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:46.984884 15858 raft_consensus.cc:504] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } }
I20260929 18:42:46.985204 15858 leader_election.cc:288] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [CANDIDATE]: Term 1 election: Requested vote from peers 7348b1e81a0b44ecb05cdc321c763613 (127.7.157.67:43983), f758aaacf28f4cb288225ed5af1dcfbd (127.7.157.65:37289)
I20260929 18:42:46.985498 15777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f63baea9982483fb6d2489259c5a58e" candidate_uuid: "da3767d8ac684ce193d63744cf6ffaf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7348b1e81a0b44ecb05cdc321c763613"
I20260929 18:42:46.985505 15497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f63baea9982483fb6d2489259c5a58e" candidate_uuid: "da3767d8ac684ce193d63744cf6ffaf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f758aaacf28f4cb288225ed5af1dcfbd"
I20260929 18:42:46.985615 15497 raft_consensus.cc:3037] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:46.985615 15777 raft_consensus.cc:3037] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:46.986754 15777 raft_consensus.cc:2454] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3767d8ac684ce193d63744cf6ffaf9 in term 1.
I20260929 18:42:46.986763 15497 raft_consensus.cc:2454] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3767d8ac684ce193d63744cf6ffaf9 in term 1.
I20260929 18:42:46.987056 15570 leader_election.cc:302] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3767d8ac684ce193d63744cf6ffaf9, f758aaacf28f4cb288225ed5af1dcfbd; no voters:
I20260929 18:42:46.987233 15858 raft_consensus.cc:2781] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:46.987414 15858 raft_consensus.cc:686] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [term 1 LEADER]: Becoming Leader. State: Replica: da3767d8ac684ce193d63744cf6ffaf9, State: Running, Role: LEADER
I20260929 18:42:46.987740 15858 consensus_queue.cc:237] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } }
I20260929 18:42:46.990732 15340 catalog_manager.cc:5495] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3767d8ac684ce193d63744cf6ffaf9 (127.7.157.66). New cstate: current_term: 1 leader_uuid: "da3767d8ac684ce193d63744cf6ffaf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } health_report { overall_health: UNKNOWN } } }
I20260929 18:42:47.085530 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.082128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.7.157.66:45709)
0929 18:42:47.082364 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:47.082368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:47.082381 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:42:47.082498 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:47.082501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:47.082598 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:47.082703 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:47.082881 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.083419 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.084270 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.084498 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.084582 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.084789 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:47.085221 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:47.085310 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:47.085362 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 18:42:47.085407 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":80,"threads_started":1}
I20260929 18:42:47.091302 15777 raft_consensus.cc:1261] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Refusing update from remote peer da3767d8ac684ce193d63744cf6ffaf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:47.091302 15497 raft_consensus.cc:1261] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd [term 1 FOLLOWER]: Refusing update from remote peer da3767d8ac684ce193d63744cf6ffaf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:47.091993 15858 consensus_queue.cc:1035] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:47.092204 15862 consensus_queue.cc:1035] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:47.100299 15870 mvcc.cc:205] Tried to move back new op lower bound from 7334737375598727168 to 7334737375184220160. Current Snapshot: MvccSnapshot[applied={T|T < 7334737375598727168}]
I20260929 18:42:47.101212 15871 mvcc.cc:205] Tried to move back new op lower bound from 7334737375598727168 to 7334737375184220160. Current Snapshot: MvccSnapshot[applied={T|T < 7334737375598727168}]
I20260929 18:42:47.108150 15338 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:47.108304 15338 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:47.110999 15338 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233: Sending DeleteTablet for 3 replicas of tablet 3f63baea9982483fb6d2489259c5a58e
I20260929 18:42:47.111881 15757 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f63baea9982483fb6d2489259c5a58e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:47 UTC) from {username='oryx', principal='oryx/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:60438
I20260929 18:42:47.111900 15617 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f63baea9982483fb6d2489259c5a58e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:47 UTC) from {username='oryx', principal='oryx/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:47874
I20260929 18:42:47.111931 15477 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f63baea9982483fb6d2489259c5a58e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:47 UTC) from {username='oryx', principal='oryx/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:51478
I20260929 18:42:47.112403 15881 tablet_replica.cc:330] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd: stopping tablet replica
I20260929 18:42:47.112398 15338 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:42:47.112588 15880 tablet_replica.cc:330] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613: stopping tablet replica
I20260929 18:42:47.112804 15881 raft_consensus.cc:2229] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:47.112926 15880 raft_consensus.cc:2229] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 18:42:47.113010 15338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:47.113207 15881 raft_consensus.cc:2258] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:47.113276 15880 raft_consensus.cc:2258] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:47.113659 15879 tablet_replica.cc:330] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9: stopping tablet replica
I20260929 18:42:47.114019 15879 raft_consensus.cc:2229] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:42:47.114450 15879 raft_consensus.cc:2258] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:47.114909 15881 ts_tablet_manager.cc:1905] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:47.116273 15880 ts_tablet_manager.cc:1905] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:47.117826 15881 ts_tablet_manager.cc:1918] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:47.117936 15881 log.cc:1196] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/wal/wals/3f63baea9982483fb6d2489259c5a58e
I20260929 18:42:47.118240 15881 ts_tablet_manager.cc:1939] T 3f63baea9982483fb6d2489259c5a58e P f758aaacf28f4cb288225ed5af1dcfbd: Deleting consensus metadata
I20260929 18:42:47.119221 15477 tablet_service.cc:1467] Processing CreateTablet for tablet bc5c4467b8d64ffabf17fbf7223a5e95 (DEFAULT_TABLE table=test-table [id=4d1bbd49ec45481aa4461c97191d47cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:47.119297 15880 ts_tablet_manager.cc:1918] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:47.119377 15880 log.cc:1196] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/3f63baea9982483fb6d2489259c5a58e
I20260929 18:42:47.119478 15477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5c4467b8d64ffabf17fbf7223a5e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:47.119608 15326 catalog_manager.cc:4841] TS f758aaacf28f4cb288225ed5af1dcfbd (127.7.157.65:37289): tablet 3f63baea9982483fb6d2489259c5a58e (table test-table [id=8fb4ec32bfc049caa6636eb1e94477d6]) successfully deleted
I20260929 18:42:47.119659 15880 ts_tablet_manager.cc:1939] T 3f63baea9982483fb6d2489259c5a58e P 7348b1e81a0b44ecb05cdc321c763613: Deleting consensus metadata
I20260929 18:42:47.120831 15879 ts_tablet_manager.cc:1905] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:47.120990 15324 catalog_manager.cc:4841] TS 7348b1e81a0b44ecb05cdc321c763613 (127.7.157.67:43983): tablet 3f63baea9982483fb6d2489259c5a58e (table test-table [id=8fb4ec32bfc049caa6636eb1e94477d6]) successfully deleted
I20260929 18:42:47.121841 15882 tablet_bootstrap.cc:492] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd: Bootstrap starting.
I20260929 18:42:47.122000 15617 tablet_service.cc:1467] Processing CreateTablet for tablet bc5c4467b8d64ffabf17fbf7223a5e95 (DEFAULT_TABLE table=test-table [id=4d1bbd49ec45481aa4461c97191d47cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:47.122259 15617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5c4467b8d64ffabf17fbf7223a5e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:47.122807 15882 tablet_bootstrap.cc:654] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:47.123377 15757 tablet_service.cc:1467] Processing CreateTablet for tablet bc5c4467b8d64ffabf17fbf7223a5e95 (DEFAULT_TABLE table=test-table [id=4d1bbd49ec45481aa4461c97191d47cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:47.123620 15757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5c4467b8d64ffabf17fbf7223a5e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:47.123624 15879 ts_tablet_manager.cc:1918] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:47.123713 15879 log.cc:1196] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/3f63baea9982483fb6d2489259c5a58e
I20260929 18:42:47.124006 15879 ts_tablet_manager.cc:1939] T 3f63baea9982483fb6d2489259c5a58e P da3767d8ac684ce193d63744cf6ffaf9: Deleting consensus metadata
I20260929 18:42:47.125840 15884 tablet_bootstrap.cc:492] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613: Bootstrap starting.
I20260929 18:42:47.125933 15324 catalog_manager.cc:4841] TS da3767d8ac684ce193d63744cf6ffaf9 (127.7.157.66:45709): tablet 3f63baea9982483fb6d2489259c5a58e (table test-table [id=8fb4ec32bfc049caa6636eb1e94477d6]) successfully deleted
I20260929 18:42:47.126914 15884 tablet_bootstrap.cc:654] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:47.127858 15885 tablet_bootstrap.cc:492] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9: Bootstrap starting.
I20260929 18:42:47.128518 15884 tablet_bootstrap.cc:492] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613: No bootstrap required, opened a new log
I20260929 18:42:47.128615 15884 ts_tablet_manager.cc:1397] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:42:47.128893 15885 tablet_bootstrap.cc:654] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:47.129020 15884 raft_consensus.cc:348] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.129109 15884 raft_consensus.cc:374] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:47.129129 15884 raft_consensus.cc:729] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7348b1e81a0b44ecb05cdc321c763613, State: Initialized, Role: FOLLOWER
I20260929 18:42:47.129242 15884 consensus_queue.cc:260] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.129478 15884 ts_tablet_manager.cc:1428] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:42:47.130195 15882 tablet_bootstrap.cc:492] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd: No bootstrap required, opened a new log
I20260929 18:42:47.130280 15882 ts_tablet_manager.cc:1397] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260929 18:42:47.130707 15882 raft_consensus.cc:348] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.130800 15882 raft_consensus.cc:374] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:47.130828 15882 raft_consensus.cc:729] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f758aaacf28f4cb288225ed5af1dcfbd, State: Initialized, Role: FOLLOWER
I20260929 18:42:47.130946 15882 consensus_queue.cc:260] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.131192 15882 ts_tablet_manager.cc:1428] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:47.138446 15885 tablet_bootstrap.cc:492] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9: No bootstrap required, opened a new log
I20260929 18:42:47.138556 15885 ts_tablet_manager.cc:1397] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260929 18:42:47.139035 15885 raft_consensus.cc:348] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.139189 15885 raft_consensus.cc:374] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:47.139230 15885 raft_consensus.cc:729] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3767d8ac684ce193d63744cf6ffaf9, State: Initialized, Role: FOLLOWER
I20260929 18:42:47.139407 15885 consensus_queue.cc:260] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.139698 15885 ts_tablet_manager.cc:1428] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:47.302825 15874 raft_consensus.cc:482] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:47.302963 15874 raft_consensus.cc:504] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.303391 15874 leader_election.cc:288] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f758aaacf28f4cb288225ed5af1dcfbd (127.7.157.65:37289), 7348b1e81a0b44ecb05cdc321c763613 (127.7.157.67:43983)
I20260929 18:42:47.303938 15497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5c4467b8d64ffabf17fbf7223a5e95" candidate_uuid: "da3767d8ac684ce193d63744cf6ffaf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" is_pre_election: true
I20260929 18:42:47.303938 15777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5c4467b8d64ffabf17fbf7223a5e95" candidate_uuid: "da3767d8ac684ce193d63744cf6ffaf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7348b1e81a0b44ecb05cdc321c763613" is_pre_election: true
I20260929 18:42:47.304114 15777 raft_consensus.cc:2454] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3767d8ac684ce193d63744cf6ffaf9 in term 0.
I20260929 18:42:47.304114 15497 raft_consensus.cc:2454] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3767d8ac684ce193d63744cf6ffaf9 in term 0.
I20260929 18:42:47.304431 15568 leader_election.cc:302] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7348b1e81a0b44ecb05cdc321c763613, da3767d8ac684ce193d63744cf6ffaf9; no voters:
I20260929 18:42:47.304636 15874 raft_consensus.cc:2781] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:47.304697 15874 raft_consensus.cc:482] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:47.304725 15874 raft_consensus.cc:3037] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:47.305711 15874 raft_consensus.cc:504] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.306026 15874 leader_election.cc:288] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [CANDIDATE]: Term 1 election: Requested vote from peers f758aaacf28f4cb288225ed5af1dcfbd (127.7.157.65:37289), 7348b1e81a0b44ecb05cdc321c763613 (127.7.157.67:43983)
I20260929 18:42:47.306402 15777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5c4467b8d64ffabf17fbf7223a5e95" candidate_uuid: "da3767d8ac684ce193d63744cf6ffaf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7348b1e81a0b44ecb05cdc321c763613"
I20260929 18:42:47.306428 15497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5c4467b8d64ffabf17fbf7223a5e95" candidate_uuid: "da3767d8ac684ce193d63744cf6ffaf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f758aaacf28f4cb288225ed5af1dcfbd"
I20260929 18:42:47.306533 15777 raft_consensus.cc:3037] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:47.306533 15497 raft_consensus.cc:3037] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:47.307667 15777 raft_consensus.cc:2454] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3767d8ac684ce193d63744cf6ffaf9 in term 1.
I20260929 18:42:47.307667 15497 raft_consensus.cc:2454] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3767d8ac684ce193d63744cf6ffaf9 in term 1.
I20260929 18:42:47.307929 15570 leader_election.cc:302] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3767d8ac684ce193d63744cf6ffaf9, f758aaacf28f4cb288225ed5af1dcfbd; no voters:
I20260929 18:42:47.308094 15874 raft_consensus.cc:2781] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:47.308192 15874 raft_consensus.cc:686] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [term 1 LEADER]: Becoming Leader. State: Replica: da3767d8ac684ce193d63744cf6ffaf9, State: Running, Role: LEADER
I20260929 18:42:47.308331 15874 consensus_queue.cc:237] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.309926 15338 catalog_manager.cc:5495] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3767d8ac684ce193d63744cf6ffaf9 (127.7.157.66). New cstate: current_term: 1 leader_uuid: "da3767d8ac684ce193d63744cf6ffaf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } health_report { overall_health: UNKNOWN } } }
Sep 29 18:42:47 dist-test-slave-smpn krb5kdc[15295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707365, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.126@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM
I20260929 18:42:47.348682 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.340809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:37083 (local address 127.0.0.1:32900)
0929 18:42:47.341159 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:47.341172 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:47.341271 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:47.341531 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:47.341534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:47.341922 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:47.342013 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:47.342019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.342738 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.342740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:47.343351 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:47.343359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.343731 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.343733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:47.343788 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.344303 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:47.344314 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:47.345749 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:47.347327 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:47.347336 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:47.347340 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:47.347592 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:47.347799 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:47.347801 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:47.347802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:47.347874 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:47.348167 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:47.348173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:47.348274 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 18:42:47.348363 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 18:42:47.348489 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":122,"threads_started":1}
I20260929 18:42:47.348686 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.340840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32900 (local address 127.7.157.126:37083)
0929 18:42:47.341201 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:47.341208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:47.341225 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:42:47.341350 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:47.341353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:47.341391 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:47.341468 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:47.342115 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.342659 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.343437 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.343648 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.343718 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.345882 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:47.345886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:47.345888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:47.345908 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:47.347219 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:47.347664 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:47.347667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:47.347668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:47.347736 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:47.347932 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:47.347934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:47.347935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:47.348098 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:47.348166 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:47.348404 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 18:42:47.348545 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":305,"thread_start_us":111,"threads_started":1}
I20260929 18:42:47.352437 15338 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:32900:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:42:47.353117 15338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:47.357551 15477 tablet_service.cc:1467] Processing CreateTablet for tablet 1d139b6eb5834965bc4bf2266a3ccc71 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50cf2061cef44a2dabbd65b103cfaa46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:42:47.357834 15757 tablet_service.cc:1467] Processing CreateTablet for tablet 1d139b6eb5834965bc4bf2266a3ccc71 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50cf2061cef44a2dabbd65b103cfaa46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:42:47.357834 15617 tablet_service.cc:1467] Processing CreateTablet for tablet 1d139b6eb5834965bc4bf2266a3ccc71 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50cf2061cef44a2dabbd65b103cfaa46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:42:47.358009 15477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d139b6eb5834965bc4bf2266a3ccc71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:47.358183 15757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d139b6eb5834965bc4bf2266a3ccc71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:47.358183 15617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d139b6eb5834965bc4bf2266a3ccc71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:47.360602 15882 tablet_bootstrap.cc:492] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd: Bootstrap starting.
I20260929 18:42:47.360772 15885 tablet_bootstrap.cc:492] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9: Bootstrap starting.
I20260929 18:42:47.360977 15884 tablet_bootstrap.cc:492] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613: Bootstrap starting.
I20260929 18:42:47.361483 15885 tablet_bootstrap.cc:654] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:47.361575 15882 tablet_bootstrap.cc:654] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:47.361913 15884 tablet_bootstrap.cc:654] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:47.362620 15885 tablet_bootstrap.cc:492] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9: No bootstrap required, opened a new log
I20260929 18:42:47.362694 15885 ts_tablet_manager.cc:1397] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:47.362910 15882 tablet_bootstrap.cc:492] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd: No bootstrap required, opened a new log
I20260929 18:42:47.362990 15882 ts_tablet_manager.cc:1397] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 18:42:47.363164 15885 raft_consensus.cc:348] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.363268 15885 raft_consensus.cc:374] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:47.363299 15885 raft_consensus.cc:729] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3767d8ac684ce193d63744cf6ffaf9, State: Initialized, Role: FOLLOWER
I20260929 18:42:47.363405 15885 consensus_queue.cc:260] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.363438 15882 raft_consensus.cc:348] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.363524 15882 raft_consensus.cc:374] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:47.363551 15882 raft_consensus.cc:729] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f758aaacf28f4cb288225ed5af1dcfbd, State: Initialized, Role: FOLLOWER
I20260929 18:42:47.363654 15885 ts_tablet_manager.cc:1428] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:47.363670 15884 tablet_bootstrap.cc:492] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613: No bootstrap required, opened a new log
I20260929 18:42:47.363653 15882 consensus_queue.cc:260] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.363735 15884 ts_tablet_manager.cc:1397] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:42:47.363838 15874 tablet_replica.cc:439] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } }
I20260929 18:42:47.363904 15882 ts_tablet_manager.cc:1428] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:42:47.363960 15874 tablet_replica.cc:442] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:47.364080 15869 tablet_replica.cc:439] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } }
I20260929 18:42:47.364187 15869 tablet_replica.cc:442] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:47.364115 15884 raft_consensus.cc:348] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.364233 15884 raft_consensus.cc:374] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:47.364267 15884 raft_consensus.cc:729] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7348b1e81a0b44ecb05cdc321c763613, State: Initialized, Role: FOLLOWER
I20260929 18:42:47.364385 15884 consensus_queue.cc:260] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.364642 15884 ts_tablet_manager.cc:1428] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:47.364840 15868 tablet_replica.cc:439] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } }
I20260929 18:42:47.364960 15868 tablet_replica.cc:442] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:47.491562 15874 raft_consensus.cc:482] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:47.491691 15874 raft_consensus.cc:504] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.492115 15874 leader_election.cc:288] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f758aaacf28f4cb288225ed5af1dcfbd (127.7.157.65:37289), 7348b1e81a0b44ecb05cdc321c763613 (127.7.157.67:43983)
I20260929 18:42:47.492564 15497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d139b6eb5834965bc4bf2266a3ccc71" candidate_uuid: "da3767d8ac684ce193d63744cf6ffaf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" is_pre_election: true
I20260929 18:42:47.492594 15777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d139b6eb5834965bc4bf2266a3ccc71" candidate_uuid: "da3767d8ac684ce193d63744cf6ffaf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7348b1e81a0b44ecb05cdc321c763613" is_pre_election: true
I20260929 18:42:47.492719 15777 raft_consensus.cc:2454] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3767d8ac684ce193d63744cf6ffaf9 in term 0.
I20260929 18:42:47.492719 15497 raft_consensus.cc:2454] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3767d8ac684ce193d63744cf6ffaf9 in term 0.
I20260929 18:42:47.493001 15568 leader_election.cc:302] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7348b1e81a0b44ecb05cdc321c763613, da3767d8ac684ce193d63744cf6ffaf9; no voters:
I20260929 18:42:47.493177 15874 raft_consensus.cc:2781] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:47.493234 15874 raft_consensus.cc:482] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:47.493284 15874 raft_consensus.cc:3037] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:47.494167 15874 raft_consensus.cc:504] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.494457 15874 leader_election.cc:288] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [CANDIDATE]: Term 1 election: Requested vote from peers f758aaacf28f4cb288225ed5af1dcfbd (127.7.157.65:37289), 7348b1e81a0b44ecb05cdc321c763613 (127.7.157.67:43983)
I20260929 18:42:47.494757 15777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d139b6eb5834965bc4bf2266a3ccc71" candidate_uuid: "da3767d8ac684ce193d63744cf6ffaf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7348b1e81a0b44ecb05cdc321c763613"
I20260929 18:42:47.494757 15497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d139b6eb5834965bc4bf2266a3ccc71" candidate_uuid: "da3767d8ac684ce193d63744cf6ffaf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f758aaacf28f4cb288225ed5af1dcfbd"
I20260929 18:42:47.494881 15777 raft_consensus.cc:3037] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:47.494889 15497 raft_consensus.cc:3037] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:47.495914 15497 raft_consensus.cc:2454] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3767d8ac684ce193d63744cf6ffaf9 in term 1.
I20260929 18:42:47.495914 15777 raft_consensus.cc:2454] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3767d8ac684ce193d63744cf6ffaf9 in term 1.
I20260929 18:42:47.496160 15568 leader_election.cc:302] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7348b1e81a0b44ecb05cdc321c763613, da3767d8ac684ce193d63744cf6ffaf9; no voters:
I20260929 18:42:47.496294 15874 raft_consensus.cc:2781] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:47.496373 15874 raft_consensus.cc:686] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [term 1 LEADER]: Becoming Leader. State: Replica: da3767d8ac684ce193d63744cf6ffaf9, State: Running, Role: LEADER
I20260929 18:42:47.496505 15874 consensus_queue.cc:237] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.497076 15858 tablet_replica.cc:439] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9: TxnStatusTablet state changed. Reason: New leader da3767d8ac684ce193d63744cf6ffaf9. Latest consensus state: current_term: 1 leader_uuid: "da3767d8ac684ce193d63744cf6ffaf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } }
I20260929 18:42:47.497161 15858 tablet_replica.cc:442] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:42:47.497362 15900 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:42:47.497550 15900 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:42:47.498098 15339 catalog_manager.cc:5495] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3767d8ac684ce193d63744cf6ffaf9 (127.7.157.66). New cstate: current_term: 1 leader_uuid: "da3767d8ac684ce193d63744cf6ffaf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } health_report { overall_health: UNKNOWN } } }
I20260929 18:42:47.516878 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.512995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:45709 (local address 127.0.0.1:47892)
0929 18:42:47.513230 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:47.513243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:47.513343 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:47.513603 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:47.513605 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:47.513610 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:42:47.513746 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:47.513751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.514794 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.514797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:47.515482 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:47.515492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.516100 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.516102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:47.516173 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.516242 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:42:47.516581 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:42:47.516595 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 18:42:47.516652 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 18:42:47.516741 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260929 18:42:47.516880 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.513079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.7.157.66:45709)
0929 18:42:47.513216 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:47.513221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:47.513285 (+ 64us) server_negotiation.cc:407] Connection header received
0929 18:42:47.513451 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:47.513454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:47.513508 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:47.513580 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:47.513839 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.514664 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.515644 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.515985 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.516088 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.516388 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:47.516462 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:47.516523 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:47.516683 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 18:42:47.516736 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:42:47.519407 15777 raft_consensus.cc:1261] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Refusing update from remote peer da3767d8ac684ce193d63744cf6ffaf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:47.519428 15497 raft_consensus.cc:1261] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd [term 1 FOLLOWER]: Refusing update from remote peer da3767d8ac684ce193d63744cf6ffaf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:47.519835 15858 consensus_queue.cc:1035] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:47.519989 15874 consensus_queue.cc:1035] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:47.521327 15869 tablet_replica.cc:439] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd: TxnStatusTablet state changed. Reason: New leader da3767d8ac684ce193d63744cf6ffaf9. Latest consensus state: current_term: 1 leader_uuid: "da3767d8ac684ce193d63744cf6ffaf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } }
I20260929 18:42:47.521440 15869 tablet_replica.cc:442] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:47.521651 15868 tablet_replica.cc:439] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613: TxnStatusTablet state changed. Reason: New leader da3767d8ac684ce193d63744cf6ffaf9. Latest consensus state: current_term: 1 leader_uuid: "da3767d8ac684ce193d63744cf6ffaf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } }
I20260929 18:42:47.521757 15868 tablet_replica.cc:442] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:47.523307 15874 tablet_replica.cc:439] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "da3767d8ac684ce193d63744cf6ffaf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } }
I20260929 18:42:47.523455 15874 tablet_replica.cc:442] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:42:47.523568 15867 mvcc.cc:205] Tried to move back new op lower bound from 7334737377355497472 to 7334737377266827264. Current Snapshot: MvccSnapshot[applied={T|T < 7334737377355497472 or (T in {7334737377355497472})}]
I20260929 18:42:47.525307 15868 tablet_replica.cc:439] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "da3767d8ac684ce193d63744cf6ffaf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } }
I20260929 18:42:47.525418 15868 tablet_replica.cc:442] T 1d139b6eb5834965bc4bf2266a3ccc71 P 7348b1e81a0b44ecb05cdc321c763613: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:47.527762 15862 tablet_replica.cc:439] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "da3767d8ac684ce193d63744cf6ffaf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } }
I20260929 18:42:47.527876 15862 tablet_replica.cc:442] T 1d139b6eb5834965bc4bf2266a3ccc71 P da3767d8ac684ce193d63744cf6ffaf9: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:42:47.529327 15869 tablet_replica.cc:439] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "da3767d8ac684ce193d63744cf6ffaf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } } }
I20260929 18:42:47.529440 15869 tablet_replica.cc:442] T 1d139b6eb5834965bc4bf2266a3ccc71 P f758aaacf28f4cb288225ed5af1dcfbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:42:47.537465 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.534261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:45709 (local address 127.7.157.66:48871)
0929 18:42:47.534548 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:47.534564 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:47.534665 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:47.534858 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:47.534861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:47.534867 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:42:47.534969 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:47.534974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.535767 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.535770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:47.536373 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:47.536380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.536804 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.536807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:47.536880 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.536920 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:42:47.537173 (+ 253us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:42:47.537178 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:42:47.537233 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 18:42:47.537323 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260929 18:42:47.537465 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.534348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:48871 (local address 127.7.157.66:45709)
0929 18:42:47.534450 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:47.534455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:47.534627 (+ 172us) server_negotiation.cc:407] Connection header received
0929 18:42:47.534742 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:47.534744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:47.534781 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:47.534837 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:47.535056 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.535671 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.536498 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.536718 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.536797 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.537025 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:47.537098 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:47.537164 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:47.537240 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 18:42:47.537310 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":40}
I20260929 18:42:47.544296 15497 raft_consensus.cc:1261] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd [term 1 FOLLOWER]: Refusing update from remote peer da3767d8ac684ce193d63744cf6ffaf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:47.544373 15777 raft_consensus.cc:1261] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Refusing update from remote peer da3767d8ac684ce193d63744cf6ffaf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:47.544656 15874 consensus_queue.cc:1035] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:47.544800 15858 consensus_queue.cc:1035] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:47.578692 15339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:47.578863 15339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:47.581027 15339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233: Sending DeleteTablet for 3 replicas of tablet bc5c4467b8d64ffabf17fbf7223a5e95
I20260929 18:42:47.581595 15477 tablet_service.cc:1514] Processing DeleteTablet for tablet bc5c4467b8d64ffabf17fbf7223a5e95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:47 UTC) from {username='oryx', principal='oryx/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:51478
I20260929 18:42:47.581637 15616 tablet_service.cc:1514] Processing DeleteTablet for tablet bc5c4467b8d64ffabf17fbf7223a5e95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:47 UTC) from {username='oryx', principal='oryx/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:47874
I20260929 18:42:47.581702 15757 tablet_service.cc:1514] Processing DeleteTablet for tablet bc5c4467b8d64ffabf17fbf7223a5e95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:47 UTC) from {username='oryx', principal='oryx/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:60438
I20260929 18:42:47.581825 15881 tablet_replica.cc:330] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd: stopping tablet replica
I20260929 18:42:47.581892 15880 tablet_replica.cc:330] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613: stopping tablet replica
I20260929 18:42:47.581907 15879 tablet_replica.cc:330] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9: stopping tablet replica
I20260929 18:42:47.581954 15881 raft_consensus.cc:2229] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:47.581987 15880 raft_consensus.cc:2229] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:47.582016 15879 raft_consensus.cc:2229] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:42:47.582072 15881 raft_consensus.cc:2258] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:47.582080 15880 raft_consensus.cc:2258] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:47.582183 15879 raft_consensus.cc:2258] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:47.583025 15879 ts_tablet_manager.cc:1905] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:47.583042 15880 ts_tablet_manager.cc:1905] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:47.585815 15880 ts_tablet_manager.cc:1918] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:42:47.585841 15879 ts_tablet_manager.cc:1918] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:42:47.585907 15879 log.cc:1196] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/bc5c4467b8d64ffabf17fbf7223a5e95
I20260929 18:42:47.586032 15880 log.cc:1196] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/bc5c4467b8d64ffabf17fbf7223a5e95
I20260929 18:42:47.586218 15879 ts_tablet_manager.cc:1939] T bc5c4467b8d64ffabf17fbf7223a5e95 P da3767d8ac684ce193d63744cf6ffaf9: Deleting consensus metadata
I20260929 18:42:47.586346 15880 ts_tablet_manager.cc:1939] T bc5c4467b8d64ffabf17fbf7223a5e95 P 7348b1e81a0b44ecb05cdc321c763613: Deleting consensus metadata
I20260929 18:42:47.586796 15881 ts_tablet_manager.cc:1905] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:47.586943 15324 catalog_manager.cc:4841] TS da3767d8ac684ce193d63744cf6ffaf9 (127.7.157.66:45709): tablet bc5c4467b8d64ffabf17fbf7223a5e95 (table test-table [id=4d1bbd49ec45481aa4461c97191d47cb]) successfully deleted
I20260929 18:42:47.587133 15324 catalog_manager.cc:4841] TS 7348b1e81a0b44ecb05cdc321c763613 (127.7.157.67:43983): tablet bc5c4467b8d64ffabf17fbf7223a5e95 (table test-table [id=4d1bbd49ec45481aa4461c97191d47cb]) successfully deleted
I20260929 18:42:47.588999 15881 ts_tablet_manager.cc:1918] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:42:47.589063 15881 log.cc:1196] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/wal/wals/bc5c4467b8d64ffabf17fbf7223a5e95
I20260929 18:42:47.589259 15881 ts_tablet_manager.cc:1939] T bc5c4467b8d64ffabf17fbf7223a5e95 P f758aaacf28f4cb288225ed5af1dcfbd: Deleting consensus metadata
I20260929 18:42:47.589951 15326 catalog_manager.cc:4841] TS f758aaacf28f4cb288225ed5af1dcfbd (127.7.157.65:37289): tablet bc5c4467b8d64ffabf17fbf7223a5e95 (table test-table [id=4d1bbd49ec45481aa4461c97191d47cb]) successfully deleted
Sep 29 18:42:47 dist-test-slave-smpn krb5kdc[15295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:42:47 dist-test-slave-smpn krb5kdc[15295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:47 dist-test-slave-smpn krb5kdc[15295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:42:47.611529 15895 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:47.604724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.7.157.126:37083)
0929 18:42:47.604849 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:47.604853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:47.604978 (+ 125us) server_negotiation.cc:407] Connection header received
0929 18:42:47.605095 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:47.605098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:47.605153 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:47.605259 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:47.605887 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.606689 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.607627 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.607970 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.608079 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.611388 (+ 3309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32906: BlockingRecv error: recv got EOF from 127.0.0.1:32906 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 18:42:47 dist-test-slave-smpn krb5kdc[15295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM
I20260929 18:42:47.622059 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.613025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.7.157.126:37083)
0929 18:42:47.613141 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:47.613145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:47.613296 (+ 151us) server_negotiation.cc:407] Connection header received
0929 18:42:47.613348 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:47.613351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:47.613402 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:47.613491 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:47.614289 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.615124 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.615904 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.616238 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.616377 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.618875 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:47.618879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:47.618882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:47.618912 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:47.620449 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:47.620830 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:47.620833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:47.620835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:47.620930 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:47.621150 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:47.621153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:47.621155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:47.621394 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:47.621531 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:47.621730 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 18:42:47.621887 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:42:47.624852 15339 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:42:47.625303 15339 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:47.629908 15477 tablet_service.cc:1467] Processing CreateTablet for tablet 8a29d5d7f2d44b3d8e565cebcf68e3e9 (DEFAULT_TABLE table=test-table [id=4a0337cfb6204be2be9066d5702d177b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:47.630044 15616 tablet_service.cc:1467] Processing CreateTablet for tablet 8a29d5d7f2d44b3d8e565cebcf68e3e9 (DEFAULT_TABLE table=test-table [id=4a0337cfb6204be2be9066d5702d177b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:47.630092 15757 tablet_service.cc:1467] Processing CreateTablet for tablet 8a29d5d7f2d44b3d8e565cebcf68e3e9 (DEFAULT_TABLE table=test-table [id=4a0337cfb6204be2be9066d5702d177b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:47.630249 15477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a29d5d7f2d44b3d8e565cebcf68e3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:47.630300 15616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a29d5d7f2d44b3d8e565cebcf68e3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:47.630359 15757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a29d5d7f2d44b3d8e565cebcf68e3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:47.632683 15882 tablet_bootstrap.cc:492] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd: Bootstrap starting.
I20260929 18:42:47.632707 15884 tablet_bootstrap.cc:492] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613: Bootstrap starting.
I20260929 18:42:47.633428 15882 tablet_bootstrap.cc:654] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:47.633510 15884 tablet_bootstrap.cc:654] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:47.634069 15885 tablet_bootstrap.cc:492] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9: Bootstrap starting.
I20260929 18:42:47.634464 15882 tablet_bootstrap.cc:492] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd: No bootstrap required, opened a new log
I20260929 18:42:47.634541 15882 ts_tablet_manager.cc:1397] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:42:47.634912 15885 tablet_bootstrap.cc:654] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:47.634907 15882 raft_consensus.cc:348] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.635015 15882 raft_consensus.cc:374] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:47.635052 15882 raft_consensus.cc:729] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f758aaacf28f4cb288225ed5af1dcfbd, State: Initialized, Role: FOLLOWER
I20260929 18:42:47.635222 15882 consensus_queue.cc:260] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.635488 15882 ts_tablet_manager.cc:1428] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:42:47.636490 15885 tablet_bootstrap.cc:492] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9: No bootstrap required, opened a new log
I20260929 18:42:47.636569 15885 ts_tablet_manager.cc:1397] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:42:47.636957 15885 raft_consensus.cc:348] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.637055 15885 raft_consensus.cc:374] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:47.637080 15885 raft_consensus.cc:729] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3767d8ac684ce193d63744cf6ffaf9, State: Initialized, Role: FOLLOWER
I20260929 18:42:47.637190 15885 consensus_queue.cc:260] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.637430 15885 ts_tablet_manager.cc:1428] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:42:47.638295 15884 tablet_bootstrap.cc:492] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613: No bootstrap required, opened a new log
I20260929 18:42:47.638397 15884 ts_tablet_manager.cc:1397] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260929 18:42:47.638867 15884 raft_consensus.cc:348] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.638983 15884 raft_consensus.cc:374] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:47.639016 15884 raft_consensus.cc:729] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7348b1e81a0b44ecb05cdc321c763613, State: Initialized, Role: FOLLOWER
I20260929 18:42:47.639163 15884 consensus_queue.cc:260] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.639451 15884 ts_tablet_manager.cc:1428] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:47.746781 15869 raft_consensus.cc:482] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:47.746918 15869 raft_consensus.cc:504] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.748014 15869 leader_election.cc:288] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da3767d8ac684ce193d63744cf6ffaf9 (127.7.157.66:45709), 7348b1e81a0b44ecb05cdc321c763613 (127.7.157.67:43983)
I20260929 18:42:47.752810 15927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.748607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:43983 (local address 127.7.157.65:33363)
0929 18:42:47.748924 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:47.748936 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:47.749030 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:47.749260 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:47.749263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:47.749277 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:47.749376 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:47.749380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.750296 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.750299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:47.751548 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:47.751562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.752378 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.752381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:47.752460 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.752481 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:42:47.752538 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 18:42:47.752607 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":50,"spinlock_wait_cycles":28032,"thread_start_us":101,"threads_started":1}
I20260929 18:42:47.752822 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.748723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:33363 (local address 127.7.157.67:43983)
0929 18:42:47.749033 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:47.749041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:47.749055 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:42:47.749100 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:47.749102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:47.749146 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:47.749220 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:47.749468 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.750191 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.751673 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.752251 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.752371 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.752423 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:47.752613 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 18:42:47.752693 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":105,"threads_started":1}
I20260929 18:42:47.753007 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.748368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:58901 (local address 127.7.157.66:45709)
0929 18:42:47.748509 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:47.748514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:47.748636 (+ 122us) server_negotiation.cc:407] Connection header received
0929 18:42:47.748784 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:47.748787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:47.748841 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:47.748911 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:42:47.749233 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.749954 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.751918 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.752489 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.752595 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.752641 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:47.752824 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:42:47.752905 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:42:47.752969 15926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.748247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:45709 (local address 127.7.157.65:58901)
0929 18:42:47.748508 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:47.748523 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:47.748669 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:47.748963 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:47.748966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:47.748984 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:42:47.749126 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:47.749132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.750061 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.750063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:47.751818 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:47.751826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.752597 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.752600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:47.752686 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.752707 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:42:47.752786 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:42:47.752851 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":25,"thread_start_us":89,"threads_started":1}
I20260929 18:42:47.753741 15776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a29d5d7f2d44b3d8e565cebcf68e3e9" candidate_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7348b1e81a0b44ecb05cdc321c763613" is_pre_election: true
I20260929 18:42:47.753775 15637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a29d5d7f2d44b3d8e565cebcf68e3e9" candidate_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3767d8ac684ce193d63744cf6ffaf9" is_pre_election: true
I20260929 18:42:47.753902 15776 raft_consensus.cc:2454] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f758aaacf28f4cb288225ed5af1dcfbd in term 0.
I20260929 18:42:47.753985 15637 raft_consensus.cc:2454] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f758aaacf28f4cb288225ed5af1dcfbd in term 0.
I20260929 18:42:47.754325 15428 leader_election.cc:302] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7348b1e81a0b44ecb05cdc321c763613, f758aaacf28f4cb288225ed5af1dcfbd; no voters:
I20260929 18:42:47.754583 15869 raft_consensus.cc:2781] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:47.754638 15869 raft_consensus.cc:482] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:47.754676 15869 raft_consensus.cc:3037] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:47.755693 15869 raft_consensus.cc:504] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.755978 15869 leader_election.cc:288] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [CANDIDATE]: Term 1 election: Requested vote from peers da3767d8ac684ce193d63744cf6ffaf9 (127.7.157.66:45709), 7348b1e81a0b44ecb05cdc321c763613 (127.7.157.67:43983)
I20260929 18:42:47.756337 15637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a29d5d7f2d44b3d8e565cebcf68e3e9" candidate_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3767d8ac684ce193d63744cf6ffaf9"
I20260929 18:42:47.756444 15637 raft_consensus.cc:3037] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:47.756686 15776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a29d5d7f2d44b3d8e565cebcf68e3e9" candidate_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7348b1e81a0b44ecb05cdc321c763613"
I20260929 18:42:47.756798 15776 raft_consensus.cc:3037] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:47.757397 15637 raft_consensus.cc:2454] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f758aaacf28f4cb288225ed5af1dcfbd in term 1.
I20260929 18:42:47.757714 15428 leader_election.cc:302] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3767d8ac684ce193d63744cf6ffaf9, f758aaacf28f4cb288225ed5af1dcfbd; no voters:
I20260929 18:42:47.757790 15776 raft_consensus.cc:2454] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f758aaacf28f4cb288225ed5af1dcfbd in term 1.
I20260929 18:42:47.757910 15869 raft_consensus.cc:2781] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:47.758121 15869 raft_consensus.cc:686] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [term 1 LEADER]: Becoming Leader. State: Replica: f758aaacf28f4cb288225ed5af1dcfbd, State: Running, Role: LEADER
I20260929 18:42:47.758360 15869 consensus_queue.cc:237] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.760633 15339 catalog_manager.cc:5495] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd reported cstate change: term changed from 0 to 1, leader changed from <none> to f758aaacf28f4cb288225ed5af1dcfbd (127.7.157.65). New cstate: current_term: 1 leader_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } health_report { overall_health: UNKNOWN } } }
I20260929 18:42:47.796710 15932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.793130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.7.157.65:37289)
0929 18:42:47.793457 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:47.793466 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:47.793481 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:42:47.793565 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:47.793568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:47.793630 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:47.793746 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:47.793973 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.794496 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.795326 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.795553 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.795624 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.795850 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:47.796304 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:47.796388 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:47.796530 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 18:42:47.796576 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":141,"threads_started":1}
I20260929 18:42:47.799463 15776 raft_consensus.cc:1261] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Refusing update from remote peer f758aaacf28f4cb288225ed5af1dcfbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:47.799592 15637 raft_consensus.cc:1261] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9 [term 1 FOLLOWER]: Refusing update from remote peer f758aaacf28f4cb288225ed5af1dcfbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:47.799930 15869 consensus_queue.cc:1035] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:47.800108 15929 consensus_queue.cc:1035] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:47.811100 15338 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:47.811277 15338 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:47.813387 15338 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233: Sending DeleteTablet for 3 replicas of tablet 8a29d5d7f2d44b3d8e565cebcf68e3e9
I20260929 18:42:47.813982 15616 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a29d5d7f2d44b3d8e565cebcf68e3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:47 UTC) from {username='oryx', principal='oryx/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:47874
I20260929 18:42:47.814167 15757 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a29d5d7f2d44b3d8e565cebcf68e3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:47 UTC) from {username='oryx', principal='oryx/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:60438
I20260929 18:42:47.814162 15477 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a29d5d7f2d44b3d8e565cebcf68e3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:47 UTC) from {username='oryx', principal='oryx/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:51478
I20260929 18:42:47.814198 15879 tablet_replica.cc:330] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9: stopping tablet replica
I20260929 18:42:47.814330 15881 tablet_replica.cc:330] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd: stopping tablet replica
I20260929 18:42:47.814330 15880 tablet_replica.cc:330] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613: stopping tablet replica
I20260929 18:42:47.814332 15879 raft_consensus.cc:2229] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:47.814427 15881 raft_consensus.cc:2229] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:42:47.814427 15880 raft_consensus.cc:2229] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:47.814464 15879 raft_consensus.cc:2258] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:47.814539 15880 raft_consensus.cc:2258] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:47.814761 15881 raft_consensus.cc:2258] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:47.815533 15881 ts_tablet_manager.cc:1905] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:47.815912 15880 ts_tablet_manager.cc:1905] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:47.815852 15338 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:42:47.816291 15338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:42:47.817958 15881 ts_tablet_manager.cc:1918] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:47.818030 15881 log.cc:1196] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-0/wal/wals/8a29d5d7f2d44b3d8e565cebcf68e3e9
I20260929 18:42:47.818178 15880 ts_tablet_manager.cc:1918] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:47.818243 15880 log.cc:1196] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/8a29d5d7f2d44b3d8e565cebcf68e3e9
I20260929 18:42:47.818271 15881 ts_tablet_manager.cc:1939] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P f758aaacf28f4cb288225ed5af1dcfbd: Deleting consensus metadata
I20260929 18:42:47.818444 15880 ts_tablet_manager.cc:1939] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P 7348b1e81a0b44ecb05cdc321c763613: Deleting consensus metadata
I20260929 18:42:47.818976 15326 catalog_manager.cc:4841] TS f758aaacf28f4cb288225ed5af1dcfbd (127.7.157.65:37289): tablet 8a29d5d7f2d44b3d8e565cebcf68e3e9 (table test-table [id=4a0337cfb6204be2be9066d5702d177b]) successfully deleted
I20260929 18:42:47.819648 15879 ts_tablet_manager.cc:1905] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:47.819727 15324 catalog_manager.cc:4841] TS 7348b1e81a0b44ecb05cdc321c763613 (127.7.157.67:43983): tablet 8a29d5d7f2d44b3d8e565cebcf68e3e9 (table test-table [id=4a0337cfb6204be2be9066d5702d177b]) successfully deleted
I20260929 18:42:47.821956 15879 ts_tablet_manager.cc:1918] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:42:47.822034 15879 log.cc:1196] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/8a29d5d7f2d44b3d8e565cebcf68e3e9
I20260929 18:42:47.822311 15879 ts_tablet_manager.cc:1939] T 8a29d5d7f2d44b3d8e565cebcf68e3e9 P da3767d8ac684ce193d63744cf6ffaf9: Deleting consensus metadata
I20260929 18:42:47.822497 15477 tablet_service.cc:1467] Processing CreateTablet for tablet 6b387a2f482540c8b2e26f278c5e63d3 (DEFAULT_TABLE table=test-table [id=581e36661d504220b359e9d511100c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:47.822753 15477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b387a2f482540c8b2e26f278c5e63d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:47.822769 15616 tablet_service.cc:1467] Processing CreateTablet for tablet 6b387a2f482540c8b2e26f278c5e63d3 (DEFAULT_TABLE table=test-table [id=581e36661d504220b359e9d511100c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:47.823009 15616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b387a2f482540c8b2e26f278c5e63d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:47.823021 15324 catalog_manager.cc:4841] TS da3767d8ac684ce193d63744cf6ffaf9 (127.7.157.66:45709): tablet 8a29d5d7f2d44b3d8e565cebcf68e3e9 (table test-table [id=4a0337cfb6204be2be9066d5702d177b]) successfully deleted
I20260929 18:42:47.823472 15757 tablet_service.cc:1467] Processing CreateTablet for tablet 6b387a2f482540c8b2e26f278c5e63d3 (DEFAULT_TABLE table=test-table [id=581e36661d504220b359e9d511100c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:42:47.823704 15757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b387a2f482540c8b2e26f278c5e63d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:47.824808 15882 tablet_bootstrap.cc:492] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd: Bootstrap starting.
I20260929 18:42:47.825670 15882 tablet_bootstrap.cc:654] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:47.825901 15884 tablet_bootstrap.cc:492] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613: Bootstrap starting.
I20260929 18:42:47.826778 15884 tablet_bootstrap.cc:654] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:47.827407 15882 tablet_bootstrap.cc:492] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd: No bootstrap required, opened a new log
I20260929 18:42:47.827490 15882 ts_tablet_manager.cc:1397] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 18:42:47.827885 15884 tablet_bootstrap.cc:492] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613: No bootstrap required, opened a new log
I20260929 18:42:47.827957 15884 ts_tablet_manager.cc:1397] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 18:42:47.827944 15882 raft_consensus.cc:348] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.828048 15882 raft_consensus.cc:374] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:47.828082 15882 raft_consensus.cc:729] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f758aaacf28f4cb288225ed5af1dcfbd, State: Initialized, Role: FOLLOWER
I20260929 18:42:47.828199 15882 consensus_queue.cc:260] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.828368 15884 raft_consensus.cc:348] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.828431 15885 tablet_bootstrap.cc:492] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9: Bootstrap starting.
I20260929 18:42:47.828457 15882 ts_tablet_manager.cc:1428] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:42:47.828459 15884 raft_consensus.cc:374] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:47.828488 15884 raft_consensus.cc:729] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7348b1e81a0b44ecb05cdc321c763613, State: Initialized, Role: FOLLOWER
I20260929 18:42:47.828598 15884 consensus_queue.cc:260] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.828830 15884 ts_tablet_manager.cc:1428] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:42:47.829166 15885 tablet_bootstrap.cc:654] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:47.830539 15885 tablet_bootstrap.cc:492] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9: No bootstrap required, opened a new log
I20260929 18:42:47.830621 15885 ts_tablet_manager.cc:1397] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:47.830978 15885 raft_consensus.cc:348] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.831118 15885 raft_consensus.cc:374] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:47.831157 15885 raft_consensus.cc:729] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3767d8ac684ce193d63744cf6ffaf9, State: Initialized, Role: FOLLOWER
I20260929 18:42:47.831285 15885 consensus_queue.cc:260] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.831522 15885 ts_tablet_manager.cc:1428] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:42:47.839977 15929 raft_consensus.cc:482] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:42:47.840101 15929 raft_consensus.cc:504] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.840472 15929 leader_election.cc:288] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da3767d8ac684ce193d63744cf6ffaf9 (127.7.157.66:45709), 7348b1e81a0b44ecb05cdc321c763613 (127.7.157.67:43983)
I20260929 18:42:47.840888 15776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b387a2f482540c8b2e26f278c5e63d3" candidate_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7348b1e81a0b44ecb05cdc321c763613" is_pre_election: true
I20260929 18:42:47.840888 15637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b387a2f482540c8b2e26f278c5e63d3" candidate_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3767d8ac684ce193d63744cf6ffaf9" is_pre_election: true
I20260929 18:42:47.841027 15776 raft_consensus.cc:2454] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f758aaacf28f4cb288225ed5af1dcfbd in term 0.
I20260929 18:42:47.841034 15637 raft_consensus.cc:2454] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f758aaacf28f4cb288225ed5af1dcfbd in term 0.
I20260929 18:42:47.841311 15428 leader_election.cc:302] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3767d8ac684ce193d63744cf6ffaf9, f758aaacf28f4cb288225ed5af1dcfbd; no voters:
I20260929 18:42:47.841485 15929 raft_consensus.cc:2781] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:42:47.841542 15929 raft_consensus.cc:482] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:42:47.841588 15929 raft_consensus.cc:3037] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:47.842787 15929 raft_consensus.cc:504] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.843164 15929 leader_election.cc:288] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [CANDIDATE]: Term 1 election: Requested vote from peers da3767d8ac684ce193d63744cf6ffaf9 (127.7.157.66:45709), 7348b1e81a0b44ecb05cdc321c763613 (127.7.157.67:43983)
I20260929 18:42:47.843480 15637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b387a2f482540c8b2e26f278c5e63d3" candidate_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3767d8ac684ce193d63744cf6ffaf9"
I20260929 18:42:47.843561 15776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b387a2f482540c8b2e26f278c5e63d3" candidate_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7348b1e81a0b44ecb05cdc321c763613"
I20260929 18:42:47.843617 15637 raft_consensus.cc:3037] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:47.843684 15776 raft_consensus.cc:3037] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:47.844702 15637 raft_consensus.cc:2454] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f758aaacf28f4cb288225ed5af1dcfbd in term 1.
I20260929 18:42:47.844702 15776 raft_consensus.cc:2454] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f758aaacf28f4cb288225ed5af1dcfbd in term 1.
I20260929 18:42:47.845010 15428 leader_election.cc:302] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7348b1e81a0b44ecb05cdc321c763613, f758aaacf28f4cb288225ed5af1dcfbd; no voters:
I20260929 18:42:47.845186 15929 raft_consensus.cc:2781] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:47.845290 15929 raft_consensus.cc:686] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [term 1 LEADER]: Becoming Leader. State: Replica: f758aaacf28f4cb288225ed5af1dcfbd, State: Running, Role: LEADER
I20260929 18:42:47.845427 15929 consensus_queue.cc:237] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } }
I20260929 18:42:47.847061 15339 catalog_manager.cc:5495] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd reported cstate change: term changed from 0 to 1, leader changed from <none> to f758aaacf28f4cb288225ed5af1dcfbd (127.7.157.65). New cstate: current_term: 1 leader_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f758aaacf28f4cb288225ed5af1dcfbd" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 } health_report { overall_health: UNKNOWN } } }
I20260929 18:42:47.870042 15943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.866449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:45709 (local address 127.7.157.65:44541)
0929 18:42:47.866746 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:47.866758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:47.866861 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:47.867115 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:47.867118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:47.867125 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:42:47.867248 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:47.867253 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.867964 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.867968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:47.868913 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:47.868924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.869340 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.869344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:47.869439 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.869485 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:42:47.869760 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:42:47.869766 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:42:47.869823 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 18:42:47.869904 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":110,"threads_started":1}
I20260929 18:42:47.870026 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.866525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:44541 (local address 127.7.157.66:45709)
0929 18:42:47.866652 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:47.866657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:47.866823 (+ 166us) server_negotiation.cc:407] Connection header received
0929 18:42:47.866955 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:47.866958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:47.867004 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:47.867087 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:47.867340 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.867863 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.869027 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.869244 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.869329 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.869613 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:47.869689 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:47.869746 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:47.869831 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 18:42:47.869882 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:42:47.875880 15637 raft_consensus.cc:1261] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9 [term 1 FOLLOWER]: Refusing update from remote peer f758aaacf28f4cb288225ed5af1dcfbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:47.875890 15776 raft_consensus.cc:1261] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Refusing update from remote peer f758aaacf28f4cb288225ed5af1dcfbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:42:47.876269 15929 consensus_queue.cc:1035] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3767d8ac684ce193d63744cf6ffaf9" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:47.876441 15869 consensus_queue.cc:1035] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7348b1e81a0b44ecb05cdc321c763613" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 43983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:42:47.893606 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.889799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:37289 (local address 127.7.157.66:56187)
0929 18:42:47.889913 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:47.889929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:47.890002 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:47.890266 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:47.890269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:47.890278 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:42:47.890369 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:47.890373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.891416 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.891419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:47.892031 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:47.892040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.892825 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.892829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:47.892915 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.892940 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:42:47.893322 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:42:47.893329 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:42:47.893386 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 18:42:47.893491 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 18:42:47.893935 15932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.889940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:56187 (local address 127.7.157.65:37289)
0929 18:42:47.890042 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:47.890046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:47.890058 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:42:47.890099 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:47.890102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:47.890146 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:47.890254 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:42:47.890462 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.891318 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.892150 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:47.892727 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:47.892818 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:47.893044 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:42:47.893130 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:42:47.893317 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:47.893772 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 18:42:47.893824 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:42:47.913101 15338 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:47.913257 15338 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:42:47.915335 15338 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f16ebedc51f46a69c11a45812442233: Sending DeleteTablet for 3 replicas of tablet 6b387a2f482540c8b2e26f278c5e63d3
I20260929 18:42:47.915874 15477 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b387a2f482540c8b2e26f278c5e63d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:47 UTC) from {username='oryx', principal='oryx/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:51478
I20260929 18:42:47.915987 15757 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b387a2f482540c8b2e26f278c5e63d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:47 UTC) from {username='oryx', principal='oryx/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:60438
I20260929 18:42:47.916131 15881 tablet_replica.cc:330] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd: stopping tablet replica
I20260929 18:42:47.916157 15880 tablet_replica.cc:330] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613: stopping tablet replica
I20260929 18:42:47.916256 15881 raft_consensus.cc:2229] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:42:47.916260 15880 raft_consensus.cc:2229] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:47.916364 15880 raft_consensus.cc:2258] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:47.916442 15881 raft_consensus.cc:2258] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:47.916633 15616 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b387a2f482540c8b2e26f278c5e63d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:42:47 UTC) from {username='oryx', principal='oryx/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:47874
I20260929 18:42:47.917019 15879 tablet_replica.cc:330] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9: stopping tablet replica
I20260929 18:42:47.917024 15881 ts_tablet_manager.cc:1905] T 6b387a2f482540c8b2e26f278c5e63d3 P f758aaacf28f4cb288225ed5af1dcfbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:47.917138 15879 raft_consensus.cc:2229] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:42:47.917259 15879 raft_consensus.cc:2258] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:42:47.918007 15879 ts_tablet_manager.cc:1905] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:47.918040 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 15412
I20260929 18:42:47.919864 15880 ts_tablet_manager.cc:1905] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:42:47.920393 15879 ts_tablet_manager.cc:1918] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:42:47.920464 15879 log.cc:1196] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/6b387a2f482540c8b2e26f278c5e63d3
I20260929 18:42:47.920686 15879 ts_tablet_manager.cc:1939] T 6b387a2f482540c8b2e26f278c5e63d3 P da3767d8ac684ce193d63744cf6ffaf9: Deleting consensus metadata
I20260929 18:42:47.921375 15324 catalog_manager.cc:4841] TS da3767d8ac684ce193d63744cf6ffaf9 (127.7.157.66:45709): tablet 6b387a2f482540c8b2e26f278c5e63d3 (table test-table [id=581e36661d504220b359e9d511100c59]) successfully deleted
I20260929 18:42:47.921941 15880 ts_tablet_manager.cc:1918] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:42:47.922006 15880 log.cc:1196] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/6b387a2f482540c8b2e26f278c5e63d3
I20260929 18:42:47.922204 15880 ts_tablet_manager.cc:1939] T 6b387a2f482540c8b2e26f278c5e63d3 P 7348b1e81a0b44ecb05cdc321c763613: Deleting consensus metadata
I20260929 18:42:47.922814 15324 catalog_manager.cc:4841] TS 7348b1e81a0b44ecb05cdc321c763613 (127.7.157.67:43983): tablet 6b387a2f482540c8b2e26f278c5e63d3 (table test-table [id=581e36661d504220b359e9d511100c59]) successfully deleted
W20260929 18:42:47.925454 15326 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:37289 (error 108)
I20260929 18:42:47.925674 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 15552
I20260929 18:42:47.927280 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:47.926749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:37289 (local address 127.0.0.1:51498)
0929 18:42:47.927094 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:47.927171 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":83,"threads_started":1}
W20260929 18:42:47.927376 15326 catalog_manager.cc:4584] TS f758aaacf28f4cb288225ed5af1dcfbd (127.7.157.65:37289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b387a2f482540c8b2e26f278c5e63d3: Network error: Client connection negotiation failed: client connection to 127.7.157.65:37289: connect: Connection refused (error 111)
I20260929 18:42:47.931888 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 15692
I20260929 18:42:47.936446 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 15308
2026-09-29T18:42:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3425 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:42:47 dist-test-slave-smpn krb5kdc[15950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:42:47 dist-test-slave-smpn krb5kdc[15950](info): set up 2 sockets
Sep 29 18:42:47 dist-test-slave-smpn krb5kdc[15950](info): commencing operation
krb5kdc: starting...
W20260929 18:42:48.883626 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:42:48 dist-test-slave-smpn krb5kdc[15950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:42:48.999770 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:41229
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:41229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:49.108374 15963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:49.108665 15963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:49.108732 15963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:49.112758 15963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:49.112825 15963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:49.112851 15963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:49.112872 15963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:49.112892 15963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:49.118312 15963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:41229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:41229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:49.119433 15963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:49.120473 15963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:49.128736 15968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:49.128834 15963 server_base.cc:1034] running on GCE node
W20260929 18:42:49.128736 15971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:49.128741 15969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:49.129664 15963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:49.130781 15963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:49.131947 15963 hybrid_clock.cc:648] HybridClock initialized: now 1790707369131908 us; error 49 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T18:42:49Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1201 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:42:49 dist-test-slave-smpn krb5kdc[15976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:42:49 dist-test-slave-smpn krb5kdc[15976](info): set up 2 sockets
Sep 29 18:42:49 dist-test-slave-smpn krb5kdc[15976](info): commencing operation
krb5kdc: starting...
W20260929 18:42:50.095465 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:42:50 dist-test-slave-smpn krb5kdc[15976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:42:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:42:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:42:50.210173 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:39795
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:39795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:50.321211 15989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:50.321481 15989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:50.321519 15989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:50.325452 15989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:50.325512 15989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:50.325531 15989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:50.325546 15989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:50.325559 15989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:50.330790 15989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:39795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:39795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:50.331847 15989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:50.332834 15989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:50.340993 15995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:50.341002 15994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:50.341114 15989 server_base.cc:1034] running on GCE node
W20260929 18:42:50.341210 15997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:50.341843 15989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:50.342969 15989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:50.344157 15989 hybrid_clock.cc:648] HybridClock initialized: now 1790707370344123 us; error 45 us; skew 500 ppm
Sep 29 18:42:50 dist-test-slave-smpn krb5kdc[15976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707370, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:50.353505 15989 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:42:50.354508 15989 webserver.cc:460] Webserver started at http://127.7.157.126:33021/ using document root <none> and password file <none>
I20260929 18:42:50.355103 15989 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:50.355183 15989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:50.355381 15989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:50.357501 15989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "c2199f7c26d445bbae164832b5d3b302"
format_stamp: "Formatted at 2026-09-29 18:42:50 on dist-test-slave-smpn"
server_key: "743d7e8d4214533f6a016477e19f7d86"
server_key_iv: "d76805f3b725a6b88d53c395e076b331"
server_key_version: "encryptionkey@0"
I20260929 18:42:50.358129 15989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "c2199f7c26d445bbae164832b5d3b302"
format_stamp: "Formatted at 2026-09-29 18:42:50 on dist-test-slave-smpn"
server_key: "743d7e8d4214533f6a016477e19f7d86"
server_key_iv: "d76805f3b725a6b88d53c395e076b331"
server_key_version: "encryptionkey@0"
I20260929 18:42:50.361963 15989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:42:50.364606 16004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:50.365886 15989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:50.365998 15989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "c2199f7c26d445bbae164832b5d3b302"
format_stamp: "Formatted at 2026-09-29 18:42:50 on dist-test-slave-smpn"
server_key: "743d7e8d4214533f6a016477e19f7d86"
server_key_iv: "d76805f3b725a6b88d53c395e076b331"
server_key_version: "encryptionkey@0"
I20260929 18:42:50.366118 15989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:50.395527 15989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:50.398499 15989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:50.398665 15989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:50.405332 15989 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:39795
I20260929 18:42:50.405357 16056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:39795 every 8 connection(s)
I20260929 18:42:50.406303 15989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:50.409109 16057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:42:50.414703 16057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302: Bootstrap starting.
I20260929 18:42:50.416195 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 15989
I20260929 18:42:50.416298 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:42:50.416551 7797 external_mini_cluster.cc:1428] Setting key 5e1754a7683e7915402b4e5dcbb557ac
I20260929 18:42:50.417342 16057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302: Neither blocks nor log segments found. Creating new log.
I20260929 18:42:50.418265 16057 log.cc:824] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:50.420248 16057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302: No bootstrap required, opened a new log
I20260929 18:42:50.423775 16057 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2199f7c26d445bbae164832b5d3b302" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39795 } }
I20260929 18:42:50.424017 16057 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:42:50.424074 16057 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2199f7c26d445bbae164832b5d3b302, State: Initialized, Role: FOLLOWER
I20260929 18:42:50.424577 16057 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2199f7c26d445bbae164832b5d3b302" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39795 } }
I20260929 18:42:50.424750 16057 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:50.424820 16057 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 29 18:42:50 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:50.424918 16057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:42:50.425827 16057 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2199f7c26d445bbae164832b5d3b302" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39795 } }
I20260929 18:42:50.426204 16057 leader_election.cc:302] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2199f7c26d445bbae164832b5d3b302; no voters:
I20260929 18:42:50.426597 16057 leader_election.cc:288] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:42:50.426715 16062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:42:50.426929 16062 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 1 LEADER]: Becoming Leader. State: Replica: c2199f7c26d445bbae164832b5d3b302, State: Running, Role: LEADER
I20260929 18:42:50.427311 16062 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2199f7c26d445bbae164832b5d3b302" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39795 } }
I20260929 18:42:50.428035 16057 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:50.428576 16063 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2199f7c26d445bbae164832b5d3b302. Latest consensus state: current_term: 1 leader_uuid: "c2199f7c26d445bbae164832b5d3b302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2199f7c26d445bbae164832b5d3b302" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39795 } } }
I20260929 18:42:50.428702 16063 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:50.429333 16064 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2199f7c26d445bbae164832b5d3b302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2199f7c26d445bbae164832b5d3b302" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39795 } } }
I20260929 18:42:50.429428 16064 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:50.431483 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:50.417697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.7.157.126:39795)
0929 18:42:50.418149 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:50.418165 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:50.418202 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:42:50.418917 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:50.418939 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:50.419332 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:50.419661 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:50.420425 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:50.421403 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:50.422232 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:50.422727 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:50.422874 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:50.425475 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:50.425481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:50.425496 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:50.425525 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:50.427583 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:50.428137 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:50.428144 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:50.428151 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:50.428298 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:50.428624 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:50.428628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:50.428630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:50.429196 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:50.429407 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:50.430345 (+ 938us) server_negotiation.cc:299] Negotiation successful
0929 18:42:50.430643 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":125,"threads_started":1}
W20260929 18:42:50.432868 16077 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:42:50.433420 16077 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:42:50.433538 16067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:50.434365 16067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:50.439944 16067 catalog_manager.cc:1347] Generated new cluster ID: 177f07626d514e3b82e8b4d2e0570999
I20260929 18:42:50.440024 16067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:50.461054 16067 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:42:50.462029 16067 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:50.465792 16067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302: Generated new TSK 0
I20260929 18:42:50.466642 16067 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:42:50.499806 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:39795
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:50.609364 16083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:50.609659 16083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:50.609720 16083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:50.613796 16083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:50.613862 16083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:50.613945 16083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:42:50.619277 16083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:39795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:50.620324 16083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:50.621327 16083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:50.630731 16091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:50.630817 16088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:50.630918 16083 server_base.cc:1034] running on GCE node
W20260929 18:42:50.630852 16089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:50.631404 16083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:50.632041 16083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:50.633194 16083 hybrid_clock.cc:648] HybridClock initialized: now 1790707370633177 us; error 29 us; skew 500 ppm
Sep 29 18:42:50 dist-test-slave-smpn krb5kdc[15976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707370, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:50.635835 16083 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:42:50.636684 16083 webserver.cc:460] Webserver started at http://127.7.157.65:42233/ using document root <none> and password file <none>
I20260929 18:42:50.637210 16083 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:50.637274 16083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:50.637458 16083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:50.639531 16083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "23c46d72ec04478d9705ab239e359ee7"
format_stamp: "Formatted at 2026-09-29 18:42:50 on dist-test-slave-smpn"
server_key: "b91939729063bc3aa0f50db2a053aba8"
server_key_iv: "1bdca37c4519b7ff75ffa755497d76d3"
server_key_version: "encryptionkey@0"
I20260929 18:42:50.640076 16083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "23c46d72ec04478d9705ab239e359ee7"
format_stamp: "Formatted at 2026-09-29 18:42:50 on dist-test-slave-smpn"
server_key: "b91939729063bc3aa0f50db2a053aba8"
server_key_iv: "1bdca37c4519b7ff75ffa755497d76d3"
server_key_version: "encryptionkey@0"
I20260929 18:42:50.643683 16083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:42:50.646081 16098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:50.647346 16083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 18:42:50.647454 16083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "23c46d72ec04478d9705ab239e359ee7"
format_stamp: "Formatted at 2026-09-29 18:42:50 on dist-test-slave-smpn"
server_key: "b91939729063bc3aa0f50db2a053aba8"
server_key_iv: "1bdca37c4519b7ff75ffa755497d76d3"
server_key_version: "encryptionkey@0"
I20260929 18:42:50.647552 16083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:50.675006 16083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:50.678021 16083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:50.678207 16083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:50.678813 16083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:50.679679 16083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:50.679729 16083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:50.679790 16083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:50.679818 16083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:50.688231 16083 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:38833
I20260929 18:42:50.688294 16211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:38833 every 8 connection(s)
I20260929 18:42:50.689146 16083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:42:50.696614 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 16083
I20260929 18:42:50.696753 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:42:50.697062 7797 external_mini_cluster.cc:1428] Setting key 93331358ba4996108adf27988a798182
Sep 29 18:42:50 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707370, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:50.702462 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:50.690943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:49931 (local address 127.7.157.126:39795)
0929 18:42:50.691131 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:50.691136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:50.691968 (+ 832us) server_negotiation.cc:407] Connection header received
0929 18:42:50.693173 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:50.693175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:50.693224 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:50.693303 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:50.694464 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:50.694976 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:50.696053 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:50.696295 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:50.696371 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:50.699164 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:50.699168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:50.699170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:50.699195 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:50.700423 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:50.700926 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:50.700929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:50.700930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:50.701007 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:50.701326 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:50.701329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:50.701331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:50.701515 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:50.701599 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:50.702129 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 18:42:50.702293 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:42:50.703368 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:50.691370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:49931)
0929 18:42:50.691814 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:50.691858 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:50.692972 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:50.693464 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:50.693471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:50.693842 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:50.694347 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:50.694360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:50.695095 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:50.695097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:50.695958 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:50.695966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:50.696462 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:50.696467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:50.696670 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:50.697287 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:50.697307 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:50.698984 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:50.700531 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:50.700537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:50.700555 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:50.700822 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:50.701084 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:50.701087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:50.701089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:50.701231 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:50.701610 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:50.701618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:50.701884 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 18:42:50.702054 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 18:42:50.702431 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
I20260929 18:42:50.704586 16212 heartbeater.cc:347] Connected to a master server at 127.7.157.126:39795
I20260929 18:42:50.704823 16212 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:50.705327 16212 heartbeater.cc:511] Master 127.7.157.126:39795 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
I20260929 18:42:50.706842 16021 ts_manager.cc:194] Registered new tserver with Master: 23c46d72ec04478d9705ab239e359ee7 (127.7.157.65:38833)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
I20260929 18:42:50.708225 16021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:49931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:42:50.722116 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:39795
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:50.829757 16217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:50.830009 16217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:50.830057 16217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:50.834136 16217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:50.834203 16217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:50.834270 16217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:42:50.839516 16217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:39795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:50.840533 16217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:50.841521 16217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:50.851011 16223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:50.851058 16217 server_base.cc:1034] running on GCE node
W20260929 18:42:50.851011 16222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:50.851032 16225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:50.851781 16217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:50.852371 16217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:50.853533 16217 hybrid_clock.cc:648] HybridClock initialized: now 1790707370853507 us; error 41 us; skew 500 ppm
Sep 29 18:42:50 dist-test-slave-smpn krb5kdc[15976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707370, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:50.856091 16217 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:42:50.856971 16217 webserver.cc:460] Webserver started at http://127.7.157.66:34935/ using document root <none> and password file <none>
I20260929 18:42:50.857497 16217 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:50.857563 16217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:50.857744 16217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:50.859833 16217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "78160dd4ace84905bcf389fcb5dabff2"
format_stamp: "Formatted at 2026-09-29 18:42:50 on dist-test-slave-smpn"
server_key: "cc812dac3a8fa29c0b7e1fafaa9e8da4"
server_key_iv: "2927b3814f50ae3619bfe71681ae928d"
server_key_version: "encryptionkey@0"
I20260929 18:42:50.860358 16217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "78160dd4ace84905bcf389fcb5dabff2"
format_stamp: "Formatted at 2026-09-29 18:42:50 on dist-test-slave-smpn"
server_key: "cc812dac3a8fa29c0b7e1fafaa9e8da4"
server_key_iv: "2927b3814f50ae3619bfe71681ae928d"
server_key_version: "encryptionkey@0"
I20260929 18:42:50.864154 16217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:42:50.866602 16232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:50.867908 16217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:50.868021 16217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "78160dd4ace84905bcf389fcb5dabff2"
format_stamp: "Formatted at 2026-09-29 18:42:50 on dist-test-slave-smpn"
server_key: "cc812dac3a8fa29c0b7e1fafaa9e8da4"
server_key_iv: "2927b3814f50ae3619bfe71681ae928d"
server_key_version: "encryptionkey@0"
I20260929 18:42:50.868125 16217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:50.893184 16217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:50.896507 16217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:50.896672 16217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:50.897209 16217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:50.898033 16217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:50.898079 16217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:50.898118 16217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:50.898135 16217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:50.906306 16217 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:39465
I20260929 18:42:50.906364 16345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:39465 every 8 connection(s)
I20260929 18:42:50.907256 16217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:42:50.907852 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 16217
I20260929 18:42:50.907986 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:42:50.908259 7797 external_mini_cluster.cc:1428] Setting key e6ab078610a588b62154358580b4a78e
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
Sep 29 18:42:50 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707370, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
I20260929 18:42:50.921527 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:50.909174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:47781 (local address 127.7.157.126:39795)
0929 18:42:50.909297 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:50.909300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:50.910359 (+ 1059us) server_negotiation.cc:407] Connection header received
0929 18:42:50.911827 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:50.911830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:50.911876 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:50.911946 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:50.913243 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:50.914059 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:50.915193 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:50.915530 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:50.915637 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:50.918100 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:50.918103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:50.918105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:50.918134 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:50.919447 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:50.919961 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:50.919963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:50.919965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:50.920041 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:50.920379 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:50.920381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:50.920382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:50.920545 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:50.920615 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:50.921228 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 18:42:50.921361 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:42:50.922492 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:50.909712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:47781)
0929 18:42:50.910208 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:50.910250 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:50.911584 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:50.912146 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:50.912156 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:50.912620 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:50.913104 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:50.913118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:50.914170 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:50.914173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:50.915050 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:50.915058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:50.915669 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:50.915671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:50.915829 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:50.916389 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:50.916405 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:50.917917 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:50.919552 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:50.919559 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:50.919576 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:50.919870 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:50.920120 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:50.920124 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:50.920126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:50.920300 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:50.920625 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:50.920632 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:50.920968 (+ 336us) client_negotiation.cc:769] Sending connection context
0929 18:42:50.921180 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 18:42:50.921510 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":142,"threads_started":1}
I20260929 18:42:50.923777 16346 heartbeater.cc:347] Connected to a master server at 127.7.157.126:39795
I20260929 18:42:50.924027 16346 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:50.924535 16346 heartbeater.cc:511] Master 127.7.157.126:39795 requested a full tablet report, sending...
I20260929 18:42:50.925750 16021 ts_manager.cc:194] Registered new tserver with Master: 78160dd4ace84905bcf389fcb5dabff2 (127.7.157.66:39465)
I20260929 18:42:50.926519 16021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:47781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:42:50.933826 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:39795
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:51.045229 16351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:51.045516 16351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:51.045575 16351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:51.050772 16351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:51.050858 16351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:51.050931 16351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:42:51.056407 16351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:39795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:51.057533 16351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:51.058527 16351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:51.068291 16356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:51.068444 16357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:51.068291 16359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:51.068532 16351 server_base.cc:1034] running on GCE node
I20260929 18:42:51.069151 16351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:51.069875 16351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:51.071054 16351 hybrid_clock.cc:648] HybridClock initialized: now 1790707371071017 us; error 49 us; skew 500 ppm
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707371, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:51.073926 16351 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:42:51.074862 16351 webserver.cc:460] Webserver started at http://127.7.157.67:37229/ using document root <none> and password file <none>
I20260929 18:42:51.075459 16351 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:51.075524 16351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:51.075738 16351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:42:51.077847 16351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "7ff633fbc0af4bceb8bc6c03abe73f38"
format_stamp: "Formatted at 2026-09-29 18:42:51 on dist-test-slave-smpn"
server_key: "f7e917bb86e2c5b2ee335c37e75f9568"
server_key_iv: "d2e00b3d9d8da50a3b0274bc39194a8e"
server_key_version: "encryptionkey@0"
I20260929 18:42:51.078420 16351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "7ff633fbc0af4bceb8bc6c03abe73f38"
format_stamp: "Formatted at 2026-09-29 18:42:51 on dist-test-slave-smpn"
server_key: "f7e917bb86e2c5b2ee335c37e75f9568"
server_key_iv: "d2e00b3d9d8da50a3b0274bc39194a8e"
server_key_version: "encryptionkey@0"
I20260929 18:42:51.082216 16351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:42:51.084729 16366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:51.085986 16351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:51.086102 16351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "7ff633fbc0af4bceb8bc6c03abe73f38"
format_stamp: "Formatted at 2026-09-29 18:42:51 on dist-test-slave-smpn"
server_key: "f7e917bb86e2c5b2ee335c37e75f9568"
server_key_iv: "d2e00b3d9d8da50a3b0274bc39194a8e"
server_key_version: "encryptionkey@0"
I20260929 18:42:51.086213 16351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:51.107939 16351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:51.111295 16351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:51.111485 16351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:51.112102 16351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:51.112948 16351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:51.112996 16351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:51.113052 16351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:51.113081 16351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:51.121569 16351 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:38955
I20260929 18:42:51.121592 16479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:38955 every 8 connection(s)
I20260929 18:42:51.122520 16351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:42:51.130313 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 16351
I20260929 18:42:51.130457 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:42:51.130755 7797 external_mini_cluster.cc:1428] Setting key ddc33d91acc8ef98c419761dcd75bf42
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707371, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:42:51.136118 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:51.124376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:60555 (local address 127.7.157.126:39795)
0929 18:42:51.124542 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:51.124547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:51.125417 (+ 870us) server_negotiation.cc:407] Connection header received
0929 18:42:51.126636 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:51.126638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:51.126690 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:51.126764 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:51.128106 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.128639 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.129959 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.130182 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.130275 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.132925 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:51.132929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:51.132932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:51.132960 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:51.134091 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:51.134584 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:51.134588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:51.134590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:51.134676 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:51.134989 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:51.134992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:51.134994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:51.135229 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:51.135316 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:51.135820 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 18:42:51.135957 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:42:51.136963 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:51.124777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:60555)
0929 18:42:51.125255 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:51.125301 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:51.126431 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:51.126938 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:51.126946 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:51.127355 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:51.127941 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.127956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.128852 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.128856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.129839 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.129851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.130343 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.130347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.130552 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.131159 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:51.131176 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:51.132776 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:42:51.134208 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:51.134214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:51.134232 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:51.134491 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:51.134757 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:42:51.134760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:42:51.134762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:42:51.134905 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:42:51.135322 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:42:51.135330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:42:51.135584 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 18:42:51.135764 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 18:42:51.136104 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":108,"threads_started":1}
I20260929 18:42:51.138080 16480 heartbeater.cc:347] Connected to a master server at 127.7.157.126:39795
I20260929 18:42:51.138310 16480 heartbeater.cc:464] Registering TS with master...
I20260929 18:42:51.138799 16480 heartbeater.cc:511] Master 127.7.157.126:39795 requested a full tablet report, sending...
I20260929 18:42:51.139851 16021 ts_manager.cc:194] Registered new tserver with Master: 7ff633fbc0af4bceb8bc6c03abe73f38 (127.7.157.67:38955)
I20260929 18:42:51.140403 16021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:60555
I20260929 18:42:51.144665 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.65.keytab.
Entry for principal oryx/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.65.keytab.
WARNING: no policy specified for oryx/127.7.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.66.keytab.
Entry for principal oryx/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.66.keytab.
WARNING: no policy specified for oryx/127.7.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.67.keytab.
Entry for principal oryx/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.67.keytab.
I20260929 18:42:51.214306 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 16083
I20260929 18:42:51.220952 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 16217
I20260929 18:42:51.225492 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 16351
I20260929 18:42:51.229912 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 15989
I20260929 18:42:51.235090 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:39795
--webserver_interface=127.7.157.126
--webserver_port=33021
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:39795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:42:51.343870 16489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:51.344151 16489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:51.344192 16489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:51.348201 16489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:42:51.348260 16489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:51.348279 16489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:51.348294 16489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:42:51.348315 16489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:42:51.353654 16489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:39795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:39795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=33021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:51.354691 16489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:51.355687 16489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:51.364043 16494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:51.364081 16495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:51.364084 16497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:51.364204 16489 server_base.cc:1034] running on GCE node
I20260929 18:42:51.365054 16489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:51.366178 16489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:51.367328 16489 hybrid_clock.cc:648] HybridClock initialized: now 1790707371367309 us; error 43 us; skew 500 ppm
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707371, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:51.369987 16489 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:42:51.370879 16489 webserver.cc:460] Webserver started at http://127.7.157.126:33021/ using document root <none> and password file <none>
I20260929 18:42:51.371474 16489 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:51.371556 16489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:51.375416 16489 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 18:42:51.377466 16504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:51.378680 16489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:51.378790 16489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "c2199f7c26d445bbae164832b5d3b302"
format_stamp: "Formatted at 2026-09-29 18:42:50 on dist-test-slave-smpn"
server_key: "743d7e8d4214533f6a016477e19f7d86"
server_key_iv: "d76805f3b725a6b88d53c395e076b331"
server_key_version: "encryptionkey@0"
I20260929 18:42:51.379374 16489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:51.397842 16489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:51.400996 16489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:51.401189 16489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:51.408299 16489 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:39795
I20260929 18:42:51.408317 16556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:39795 every 8 connection(s)
I20260929 18:42:51.409394 16489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:42:51.410563 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 16489
I20260929 18:42:51.411032 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:38833
--local_ip_for_outbound_sockets=127.7.157.65
--tserver_master_addrs=127.7.157.126:39795
--webserver_port=42233
--webserver_interface=127.7.157.65
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.157.65
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:42:51.414526 16557 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:42:51.416385 16557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302: Bootstrap starting.
I20260929 18:42:51.433024 16557 log.cc:824] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302: Log is configured to *not* fsync() on all Append() calls
I20260929 18:42:51.439963 16557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:42:51.440331 16557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302: Bootstrap complete.
I20260929 18:42:51.444484 16557 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2199f7c26d445bbae164832b5d3b302" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39795 } }
I20260929 18:42:51.444914 16557 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2199f7c26d445bbae164832b5d3b302, State: Initialized, Role: FOLLOWER
I20260929 18:42:51.445459 16557 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2199f7c26d445bbae164832b5d3b302" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39795 } }
I20260929 18:42:51.445597 16557 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:42:51.445668 16557 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:42:51.445761 16557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:42:51.447721 16557 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2199f7c26d445bbae164832b5d3b302" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39795 } }
I20260929 18:42:51.448024 16557 leader_election.cc:302] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2199f7c26d445bbae164832b5d3b302; no voters:
I20260929 18:42:51.448324 16557 leader_election.cc:288] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:42:51.448443 16561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:42:51.448686 16561 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [term 2 LEADER]: Becoming Leader. State: Replica: c2199f7c26d445bbae164832b5d3b302, State: Running, Role: LEADER
I20260929 18:42:51.449031 16561 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2199f7c26d445bbae164832b5d3b302" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39795 } }
I20260929 18:42:51.449440 16557 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:42:51.449991 16561 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2199f7c26d445bbae164832b5d3b302. Latest consensus state: current_term: 2 leader_uuid: "c2199f7c26d445bbae164832b5d3b302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2199f7c26d445bbae164832b5d3b302" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39795 } } }
I20260929 18:42:51.450181 16561 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:51.450371 16562 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c2199f7c26d445bbae164832b5d3b302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2199f7c26d445bbae164832b5d3b302" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39795 } } }
I20260929 18:42:51.450536 16562 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302 [sys.catalog]: This master's current role is: LEADER
I20260929 18:42:51.450584 16569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:42:51.453850 16569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:42:51.454530 16569 catalog_manager.cc:1259] Loaded cluster ID: 177f07626d514e3b82e8b4d2e0570999
I20260929 18:42:51.454582 16569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:42:51.456275 16569 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:42:51.456823 16569 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c2199f7c26d445bbae164832b5d3b302: Loaded TSK: 0
I20260929 18:42:51.457784 16569 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:42:51.534389 16559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:51.534646 16559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:51.534715 16559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:51.538762 16559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:51.538825 16559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:51.538905 16559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:42:51.544184 16559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.65.keytab
--principal=oryx/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:38833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=42233
--tserver_master_addrs=127.7.157.126:39795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:51.545189 16559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:51.546201 16559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:51.555840 16584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:51.555828 16583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:51.555828 16586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:51.556393 16559 server_base.cc:1034] running on GCE node
I20260929 18:42:51.556789 16559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:51.557410 16559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:51.558579 16559 hybrid_clock.cc:648] HybridClock initialized: now 1790707371558545 us; error 47 us; skew 500 ppm
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707371, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:51.561133 16559 init.cc:377] Logged in from keytab as oryx/127.7.157.65@KRBTEST.COM (short username oryx)
I20260929 18:42:51.561981 16559 webserver.cc:460] Webserver started at http://127.7.157.65:42233/ using document root <none> and password file <none>
I20260929 18:42:51.562515 16559 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:51.562585 16559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:51.566471 16559 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:42:51.568536 16593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:51.569789 16559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:42:51.569895 16559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "23c46d72ec04478d9705ab239e359ee7"
format_stamp: "Formatted at 2026-09-29 18:42:50 on dist-test-slave-smpn"
server_key: "b91939729063bc3aa0f50db2a053aba8"
server_key_iv: "1bdca37c4519b7ff75ffa755497d76d3"
server_key_version: "encryptionkey@0"
I20260929 18:42:51.570420 16559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:51.584302 16559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:51.587350 16559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:51.587522 16559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:51.588115 16559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:51.588992 16559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:51.589038 16559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:51.589078 16559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:51.589105 16559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:51.598614 16559 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:38833
I20260929 18:42:51.598634 16706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:38833 every 8 connection(s)
I20260929 18:42:51.599658 16559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:42:51.607636 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 16559
I20260929 18:42:51.608047 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:39465
--local_ip_for_outbound_sockets=127.7.157.66
--tserver_master_addrs=127.7.157.126:39795
--webserver_port=34935
--webserver_interface=127.7.157.66
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.157.66
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:42:51.612263 16709 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:51.601709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:50617 (local address 127.7.157.126:39795)
0929 18:42:51.602189 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:51.602209 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:51.602425 (+ 216us) server_negotiation.cc:407] Connection header received
0929 18:42:51.603840 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:51.603852 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:51.604150 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:51.604427 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:51.605465 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.606452 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.607576 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.608070 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.608217 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.611541 (+ 3324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:50617: BlockingRecv error: recv got EOF from 127.7.157.65:50617 (error 108)
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":94,"threads_started":1}
I20260929 18:42:51.612286 16710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:51.601833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:50617)
0929 18:42:51.602265 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:51.602322 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:51.603459 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:51.604496 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:51.604502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:51.604853 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:51.605300 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.605313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.606551 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.606554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.607451 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.607459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.608222 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.608226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.608396 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.608948 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:51.608964 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:51.611321 (+ 2357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":144,"threads_started":1}
W20260929 18:42:51.612769 16707 heartbeater.cc:644] Failed to heartbeat to 127.7.157.126:39795 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.157.126:39795: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:51.617961 16710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:51.613169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:33545)
0929 18:42:51.613315 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:51.613328 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:51.613398 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:51.613614 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:51.613617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:51.613758 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:51.613850 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.613855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.614565 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.614569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.615148 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.615157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.615567 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.615570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.615655 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.616093 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:51.616105 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:51.617838 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90}
W20260929 18:42:51.618300 16709 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:51.613276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:33545 (local address 127.7.157.126:39795)
0929 18:42:51.613393 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:51.613397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:51.613409 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:42:51.613483 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:51.613485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:51.613529 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:51.613591 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:51.613953 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.614470 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.615251 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.615472 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.615557 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.617968 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:33545: BlockingRecv error: recv got EOF from 127.7.157.65:33545 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:51.623410 16710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:51.618467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:38775)
0929 18:42:51.618583 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:51.618596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:51.618664 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:51.619004 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:51.619006 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:51.619150 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:51.619220 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.619225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.620164 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.620166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.620660 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.620666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.621164 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.621166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.621218 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.621614 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:51.621625 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:51.623314 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:42:51.623595 16707 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:42:51.623693 16709 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:51.618704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:38775 (local address 127.7.157.126:39795)
0929 18:42:51.618819 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:51.618823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:51.618836 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:42:51.618870 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:51.618873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:51.618914 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:51.618989 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:51.619312 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.620082 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.620745 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.621083 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.621172 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.623462 (+ 2290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:38775: BlockingRecv error: recv got EOF from 127.7.157.65:38775 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 18:42:51.719219 16711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:51.719498 16711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:51.719540 16711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:51.723562 16711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:51.723624 16711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:51.723690 16711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:42:51.729004 16711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.66.keytab
--principal=oryx/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:39465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=34935
--tserver_master_addrs=127.7.157.126:39795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:51.730059 16711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:51.731046 16711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:51.740377 16719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:51.740450 16711 server_base.cc:1034] running on GCE node
W20260929 18:42:51.740600 16716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:51.740509 16717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:51.741039 16711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:51.741674 16711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:51.742826 16711 hybrid_clock.cc:648] HybridClock initialized: now 1790707371742811 us; error 31 us; skew 500 ppm
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707371, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:51.745510 16711 init.cc:377] Logged in from keytab as oryx/127.7.157.66@KRBTEST.COM (short username oryx)
I20260929 18:42:51.746371 16711 webserver.cc:460] Webserver started at http://127.7.157.66:34935/ using document root <none> and password file <none>
I20260929 18:42:51.746889 16711 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:51.746963 16711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:51.751134 16711 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:42:51.753244 16726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:51.754469 16711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:51.754572 16711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "78160dd4ace84905bcf389fcb5dabff2"
format_stamp: "Formatted at 2026-09-29 18:42:50 on dist-test-slave-smpn"
server_key: "cc812dac3a8fa29c0b7e1fafaa9e8da4"
server_key_iv: "2927b3814f50ae3619bfe71681ae928d"
server_key_version: "encryptionkey@0"
I20260929 18:42:51.755141 16711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:51.772905 16711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:51.775801 16711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:51.775969 16711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:51.776677 16711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:51.777515 16711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:51.777572 16711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:51.777626 16711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:51.777649 16711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:51.786074 16711 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:39465
I20260929 18:42:51.786099 16839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:39465 every 8 connection(s)
I20260929 18:42:51.787053 16711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:42:51.794680 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 16711
I20260929 18:42:51.795117 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:38955
--local_ip_for_outbound_sockets=127.7.157.67
--tserver_master_addrs=127.7.157.126:39795
--webserver_port=37229
--webserver_interface=127.7.157.67
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.157.67
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:42:51.799024 16709 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:51.788851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:34449 (local address 127.7.157.126:39795)
0929 18:42:51.788982 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:51.788986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:51.791005 (+ 2019us) server_negotiation.cc:407] Connection header received
0929 18:42:51.792289 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:51.792292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:51.792345 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:51.792422 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:51.793535 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.794069 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.795237 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.795530 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.795641 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.798888 (+ 3247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:34449: BlockingRecv error: recv got EOF from 127.7.157.66:34449 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:42:51.799580 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:51.789320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:34449)
0929 18:42:51.790866 (+ 1546us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:51.790907 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:51.792067 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:51.792601 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:51.792608 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:51.792950 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:51.793415 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.793428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.794163 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.794166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.795114 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.795123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.795727 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.795731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.795877 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.796467 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:51.796483 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:51.798733 (+ 2250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1364,"thread_start_us":109,"threads_started":1}
W20260929 18:42:51.800148 16840 heartbeater.cc:644] Failed to heartbeat to 127.7.157.126:39795 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.157.126:39795: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:51.805212 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:51.800601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:37211)
0929 18:42:51.800701 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:51.800715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:51.800791 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:51.801008 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:51.801011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:51.801167 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:51.801256 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.801260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.801953 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.801956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.802508 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.802517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.802924 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.802927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.803010 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.803476 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:51.803489 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:51.805070 (+ 1581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 18:42:51.805306 16709 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:51.800700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:37211 (local address 127.7.157.126:39795)
0929 18:42:51.800806 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:51.800811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:51.800841 (+ 30us) server_negotiation.cc:407] Connection header received
0929 18:42:51.800882 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:51.800885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:51.800921 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:51.800981 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:51.801360 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.801856 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.802613 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.802830 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.802912 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.805206 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:37211: BlockingRecv error: recv got EOF from 127.7.157.66:37211 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:51.811089 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:51.806139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:59431)
0929 18:42:51.806301 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:51.806314 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:51.806376 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:51.806586 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:51.806588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:51.806706 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:51.806775 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.806779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.807793 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.807795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.808311 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.808319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.808865 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.808868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.808926 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.809338 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:51.809350 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:51.810934 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20260929 18:42:51.811187 16709 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:51.806233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:59431 (local address 127.7.157.126:39795)
0929 18:42:51.806348 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:51.806352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:51.806366 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:42:51.806439 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:51.806442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:51.806485 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:51.806559 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:51.806855 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.807684 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.808403 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.808767 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.808854 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.811043 (+ 2189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:59431: BlockingRecv error: recv got EOF from 127.7.157.66:59431 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 18:42:51.811370 16840 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:42:51.908074 16843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:42:51.908354 16843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:42:51.908414 16843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:42:51.912421 16843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:42:51.912482 16843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:42:51.912560 16843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:42:51.917925 16843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/krb5kdc/oryx_127.7.157.67.keytab
--principal=oryx/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:38955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=37229
--tserver_master_addrs=127.7.157.126:39795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:42:51.919023 16843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:42:51.920104 16843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:42:51.929559 16848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:51.929533 16851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:42:51.929533 16849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:42:51.929569 16843 server_base.cc:1034] running on GCE node
I20260929 18:42:51.930143 16843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:42:51.930778 16843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:42:51.931926 16843 hybrid_clock.cc:648] HybridClock initialized: now 1790707371931901 us; error 35 us; skew 500 ppm
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707371, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:42:51.934525 16843 init.cc:377] Logged in from keytab as oryx/127.7.157.67@KRBTEST.COM (short username oryx)
I20260929 18:42:51.935420 16843 webserver.cc:460] Webserver started at http://127.7.157.67:37229/ using document root <none> and password file <none>
I20260929 18:42:51.935948 16843 fs_manager.cc:359] Metadata directory not provided
I20260929 18:42:51.936018 16843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:42:51.939733 16843 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:42:51.941675 16858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:51.942883 16843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:42:51.942986 16843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "7ff633fbc0af4bceb8bc6c03abe73f38"
format_stamp: "Formatted at 2026-09-29 18:42:51 on dist-test-slave-smpn"
server_key: "f7e917bb86e2c5b2ee335c37e75f9568"
server_key_iv: "d2e00b3d9d8da50a3b0274bc39194a8e"
server_key_version: "encryptionkey@0"
I20260929 18:42:51.943536 16843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:42:51.962800 16843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:42:51.965795 16843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:42:51.965970 16843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:42:51.966554 16843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:42:51.967401 16843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:42:51.967448 16843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:51.967500 16843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:42:51.967525 16843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:42:51.975895 16843 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:38955
I20260929 18:42:51.975924 16971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:38955 every 8 connection(s)
I20260929 18:42:51.976920 16843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:42:51.981432 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 16843
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:42:51.988966 16709 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:51.978827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:44249 (local address 127.7.157.126:39795)
0929 18:42:51.978960 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:51.978964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:51.979820 (+ 856us) server_negotiation.cc:407] Connection header received
0929 18:42:51.981090 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:51.981094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:51.981152 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:51.981255 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:51.982701 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.983483 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.984962 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.985308 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.985411 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.988835 (+ 3424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:44249: BlockingRecv error: recv got EOF from 127.7.157.67:44249 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:42:51.989617 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:51.979237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:44249)
0929 18:42:51.979654 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:51.979697 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:51.980880 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:51.981442 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:51.981450 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:51.981839 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:51.982527 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.982547 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.983598 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.983601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.984845 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.984854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.985474 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.985479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.985701 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.986320 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:51.986336 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:51.988697 (+ 2361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":98,"threads_started":1}
W20260929 18:42:51.990075 16972 heartbeater.cc:644] Failed to heartbeat to 127.7.157.126:39795 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.157.126:39795: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
I20260929 18:42:51.991281 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:51.982404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.7.157.126:39795)
0929 18:42:51.982688 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:51.982691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:51.982701 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:42:51.982735 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:51.982737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:51.982773 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:51.982839 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:51.983695 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.984478 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.985487 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.985804 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.985913 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.987196 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:42:51.987202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:42:51.987221 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:42:51.987255 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:42:51.988953 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:51.989408 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:51.989413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:51.989418 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:51.989535 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:42:51.989850 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:42:51.989853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:42:51.989855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:42:51.990288 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:42:51.990476 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 18:42:51.990878 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 18:42:51.991133 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":85,"threads_started":1}
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:51.995492 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:51.990482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:49393)
0929 18:42:51.990584 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:51.990597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:51.990655 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:51.990850 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:51.990852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:51.990997 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:51.991129 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.991136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.991900 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.991903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.992667 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.992677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.993234 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.993237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.993316 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.993765 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:51.993777 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:51.995401 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 18:42:51.995683 16709 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:51.990549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:49393 (local address 127.7.157.126:39795)
0929 18:42:51.990649 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:51.990653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:51.990666 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:42:51.990705 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:51.990707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:51.990749 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:51.990818 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:51.991221 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.991801 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.992796 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.993119 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.993227 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.995530 (+ 2303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:49393: BlockingRecv error: recv got EOF from 127.7.157.67:49393 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:51 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:51.999998 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:51.995939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:38455)
0929 18:42:51.996038 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:51.996051 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:51.996110 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:51.996381 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:51.996383 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:51.996493 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:51.996558 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.996562 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.997177 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.997179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.997677 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:51.997683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.998022 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.998025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:51.998075 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.998414 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:51.998424 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:51.999880 (+ 1456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 18:42:52.000087 16709 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:51.996099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:38455 (local address 127.7.157.126:39795)
0929 18:42:51.996219 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:51.996223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:51.996234 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:42:51.996263 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:51.996264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:51.996297 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:51.996364 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:51.996632 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.997108 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.997754 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:51.997949 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:51.998018 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:51.999995 (+ 1977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:38455: BlockingRecv error: recv got EOF from 127.7.157.67:38455 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 18:42:52.000216 16972 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 18:42:52 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:52 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:52.629369 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:52.624212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:52961)
0929 18:42:52.624466 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:52.624479 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:52.624573 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:52.624840 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:52.624843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:52.625064 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:52.625189 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:52.625198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:52.625916 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:52.625918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:52.626477 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:52.626484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:52.626910 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:52.626912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:52.626972 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:52.627424 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:52.627435 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:52.629240 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
W20260929 18:42:52.629485 16978 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:52.624288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:52961 (local address 127.7.157.126:39795)
0929 18:42:52.624517 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:52.624522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:52.624535 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:42:52.624664 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:52.624666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:52.624709 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:52.624798 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:52.625306 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:52.625831 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:52.626591 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:52.626823 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:52.626900 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:52.629370 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:52961: BlockingRecv error: recv got EOF from 127.7.157.65:52961 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
Sep 29 18:42:52 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:52 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:52.817077 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:52.811948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:56499)
0929 18:42:52.812242 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:52.812255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:52.812351 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:52.812539 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:52.812541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:52.812705 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:52.812805 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:52.812811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:52.813611 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:52.813613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:52.814193 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:52.814203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:52.814616 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:52.814618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:52.814675 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:52.815135 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:52.815146 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:52.816953 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":80,"threads_started":1}
W20260929 18:42:52.817183 16978 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:52.812006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:56499 (local address 127.7.157.126:39795)
0929 18:42:52.812172 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:52.812176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:52.812319 (+ 143us) server_negotiation.cc:407] Connection header received
0929 18:42:52.812419 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:52.812421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:52.812464 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:52.812527 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:52.812938 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:52.813478 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:52.814303 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:52.814528 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:52.814603 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:52.817087 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:56499: BlockingRecv error: recv got EOF from 127.7.157.66:56499 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 18:42:53 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:53 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:53.005970 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:53.000791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:34485)
0929 18:42:53.001027 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:53.001039 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:53.001137 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:53.001341 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:53.001343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:53.001508 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:53.001600 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:53.001605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:53.002367 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:53.002369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:53.002927 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:53.002933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:53.003360 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:53.003362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:53.003424 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:53.003917 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:53.003928 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:53.005841 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":67,"threads_started":1}
W20260929 18:42:53.006181 16978 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:53.000886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:34485 (local address 127.7.157.126:39795)
0929 18:42:53.001002 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:53.001006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:53.001103 (+ 97us) server_negotiation.cc:407] Connection header received
0929 18:42:53.001207 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:53.001209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:53.001251 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:53.001317 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:53.001731 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:53.002265 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:53.003016 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:53.003264 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:53.003336 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:53.005996 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:34485: BlockingRecv error: recv got EOF from 127.7.157.67:34485 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 18:42:53 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:53 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:53.635217 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:53.630141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:38195)
0929 18:42:53.630388 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:53.630403 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:53.630491 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:53.630708 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:53.630710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:53.630896 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:53.630993 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:53.630998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:53.631810 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:53.631813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:53.632369 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:53.632377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:53.632792 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:53.632795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:53.632853 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:53.633331 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:53.633341 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:53.635040 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":66,"threads_started":1}
W20260929 18:42:53.635382 16982 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:53.630254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:38195 (local address 127.7.157.126:39795)
0929 18:42:53.630479 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:53.630483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:53.630495 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:42:53.630573 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:53.630575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:53.630616 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:53.630683 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:53.631129 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:53.631680 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:53.632455 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:53.632666 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:53.632774 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:53.635224 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:38195: BlockingRecv error: recv got EOF from 127.7.157.65:38195 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":78,"threads_started":1}
Sep 29 18:42:53 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:53 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:53.822949 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:53.817777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:38963)
0929 18:42:53.818051 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:53.818065 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:53.818175 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:53.818431 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:53.818434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:53.818625 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:53.818718 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:53.818723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:53.819505 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:53.819508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:53.820063 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:53.820071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:53.820508 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:53.820511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:53.820588 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:53.821052 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:53.821063 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:53.822812 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
W20260929 18:42:53.823015 16982 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:53.817845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:38963 (local address 127.7.157.126:39795)
0929 18:42:53.818021 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:53.818026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:53.818139 (+ 113us) server_negotiation.cc:407] Connection header received
0929 18:42:53.818282 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:53.818285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:53.818332 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:53.818397 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:53.818830 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:53.819409 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:53.820187 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:53.820410 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:53.820494 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:53.822922 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:38963: BlockingRecv error: recv got EOF from 127.7.157.66:38963 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 18:42:54 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:54 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:54.011853 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:54.006686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:57977)
0929 18:42:54.006984 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:54.006997 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:54.007106 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:54.007331 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:54.007333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:54.007506 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:54.007604 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:54.007609 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:54.008331 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:54.008334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:54.008880 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:54.008887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:54.009298 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:54.009301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:54.009361 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:54.009831 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:54.009842 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:54.011666 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":116,"threads_started":1}
W20260929 18:42:54.011934 16982 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:54.006858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:57977 (local address 127.7.157.126:39795)
0929 18:42:54.007028 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:54.007032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:54.007047 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:42:54.007184 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:54.007186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:54.007233 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:54.007305 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:54.007726 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:54.008250 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:54.008980 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:54.009204 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:54.009278 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:54.011822 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:57977: BlockingRecv error: recv got EOF from 127.7.157.67:57977 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 18:42:54 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:54 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:54.641368 16985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:54.636048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:60825)
0929 18:42:54.636311 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:54.636325 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:54.636416 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:54.636682 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:54.636684 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:54.636870 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:54.636965 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:54.636971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:54.637686 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:54.637688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:54.638243 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:54.638250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:54.638761 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:54.638765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:54.638852 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:54.639444 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:54.639456 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:54.641223 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":83,"threads_started":1}
W20260929 18:42:54.641583 16986 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:54.636152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:60825 (local address 127.7.157.126:39795)
0929 18:42:54.636390 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:54.636393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:54.636407 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:42:54.636500 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:54.636502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:54.636546 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:54.636655 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:54.637052 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:54.637570 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:54.638372 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:54.638610 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:54.638729 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:54.641375 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:60825: BlockingRecv error: recv got EOF from 127.7.157.65:60825 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":59,"threads_started":1}
Sep 29 18:42:54 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:54 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:54.829219 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:54.823888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:43507)
0929 18:42:54.824221 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:54.824235 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:54.824367 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:54.824641 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:54.824644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:54.824863 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:54.824988 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:54.824996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:54.825886 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:54.825888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:54.826446 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:54.826454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:54.826878 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:54.826880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:54.826939 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:54.827452 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:54.827468 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:54.829102 (+ 1634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":73,"threads_started":1}
W20260929 18:42:54.829404 16986 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:54.824008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:43507 (local address 127.7.157.126:39795)
0929 18:42:54.824192 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:54.824198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:54.824333 (+ 135us) server_negotiation.cc:407] Connection header received
0929 18:42:54.824465 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:54.824468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:54.824524 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:54.824614 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:54.825124 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:54.825782 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:54.826556 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:54.826786 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:54.826875 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:54.829273 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:43507: BlockingRecv error: recv got EOF from 127.7.157.66:43507 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 18:42:55 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:55 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:55.017525 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:55.012660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:41123)
0929 18:42:55.012900 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:55.012912 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:55.013006 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:55.013207 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:55.013209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:55.013380 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:55.013475 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:55.013480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:55.014241 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:55.014244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:55.014795 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:55.014802 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:55.015232 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:55.015234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:55.015292 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:55.015760 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:55.015770 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:55.017412 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":67,"threads_started":1}
W20260929 18:42:55.017724 16986 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:55.012722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:41123 (local address 127.7.157.126:39795)
0929 18:42:55.012846 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:55.012850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:55.012971 (+ 121us) server_negotiation.cc:407] Connection header received
0929 18:42:55.013072 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:55.013074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:55.013121 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:55.013186 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:55.013635 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:55.014151 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:55.014905 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:55.015150 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:55.015227 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:55.017576 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:41123: BlockingRecv error: recv got EOF from 127.7.157.67:41123 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 18:42:55 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:55 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:55.652347 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:55.642163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:59553)
0929 18:42:55.642442 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:55.642456 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:55.642540 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:55.642771 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:55.642773 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:55.642937 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:55.643032 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:55.643037 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:55.643833 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:55.643835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:55.644389 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:55.644396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:55.644840 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:55.644842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:55.644900 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:55.650407 (+ 5507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:55.650420 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:55.652182 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
W20260929 18:42:55.652473 16990 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:55.642215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:59553 (local address 127.7.157.126:39795)
0929 18:42:55.642513 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:55.642523 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:55.642538 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:42:55.642622 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:55.642624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:55.642669 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:55.642750 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:55.643172 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:55.643741 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:55.644490 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:55.644752 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:55.644830 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:55.652348 (+ 7518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:59553: BlockingRecv error: recv got EOF from 127.7.157.65:59553 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":113,"threads_started":1}
Sep 29 18:42:55 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:55 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:55.835278 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:55.830090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:38435)
0929 18:42:55.830381 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:55.830394 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:55.830495 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:55.830730 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:55.830732 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:55.830908 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:55.831005 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:55.831010 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:55.831784 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:55.831787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:55.832337 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:55.832345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:55.832755 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:55.832758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:55.832828 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:55.833302 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:55.833314 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:55.835157 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":85,"threads_started":1}
W20260929 18:42:55.835387 16990 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:55.830182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:38435 (local address 127.7.157.126:39795)
0929 18:42:55.830334 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:55.830338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:55.830467 (+ 129us) server_negotiation.cc:407] Connection header received
0929 18:42:55.830577 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:55.830579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:55.830623 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:55.830679 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:55.831120 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:55.831657 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:55.832430 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:55.832636 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:55.832707 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:55.835279 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:38435: BlockingRecv error: recv got EOF from 127.7.157.66:38435 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 18:42:56 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:56 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:56.023658 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:56.018321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:54805)
0929 18:42:56.018586 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:56.018599 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:56.018710 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:56.018977 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:56.018979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:56.019199 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:56.019309 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:56.019316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:56.020067 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:56.020071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:56.020647 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:56.020655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:56.021119 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:56.021123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:56.021204 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:56.021737 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:56.021749 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:56.023512 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":98,"threads_started":1}
W20260929 18:42:56.023748 16990 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:56.018440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:54805 (local address 127.7.157.126:39795)
0929 18:42:56.018578 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:56.018583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:56.018670 (+ 87us) server_negotiation.cc:407] Connection header received
0929 18:42:56.018821 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:56.018824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:56.018873 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:56.018938 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:56.019438 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:56.019965 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:56.020759 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:56.021027 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:56.021116 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:56.023651 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:54805: BlockingRecv error: recv got EOF from 127.7.157.67:54805 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 18:42:56 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:56 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:56.658389 16993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:56.653269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:45201)
0929 18:42:56.653550 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:56.653564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:56.653660 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:56.653910 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:56.653912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:56.654098 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:56.654203 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:56.654209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:56.654975 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:56.654977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:56.655586 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:56.655594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:56.656002 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:56.656004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:56.656070 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:56.656534 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:56.656544 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:56.658269 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":70,"threads_started":1}
W20260929 18:42:56.658600 16994 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:56.653375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:45201 (local address 127.7.157.126:39795)
0929 18:42:56.653591 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:56.653595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:56.653611 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:42:56.653740 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:56.653742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:56.653784 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:56.653865 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:56.654323 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:56.654869 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:56.655680 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:56.655905 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:56.655977 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:56.658409 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:45201: BlockingRecv error: recv got EOF from 127.7.157.65:45201 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":63,"threads_started":1}
Sep 29 18:42:56 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:56 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:56.841208 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:56.836036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:33949)
0929 18:42:56.836307 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:56.836322 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:56.836423 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:56.836661 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:56.836664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:56.836867 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:56.836959 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:56.836966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:56.837703 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:56.837707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:56.838259 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:56.838267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:56.838689 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:56.838692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:56.838771 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:56.839250 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:56.839261 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:56.841059 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":93,"threads_started":1}
W20260929 18:42:56.841317 16994 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:56.836229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:33949 (local address 127.7.157.126:39795)
0929 18:42:56.836343 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:56.836347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:56.836377 (+ 30us) server_negotiation.cc:407] Connection header received
0929 18:42:56.836510 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:56.836513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:56.836559 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:56.836622 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:56.837070 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:56.837593 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:56.838369 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:56.838591 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:56.838675 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:56.841220 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:33949: BlockingRecv error: recv got EOF from 127.7.157.66:33949 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 29 18:42:57 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:57 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:57.029805 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:57.024490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:43997)
0929 18:42:57.024768 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:57.024784 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:57.024899 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:57.025165 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:57.025168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:57.025366 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:57.025459 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:57.025466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:57.026225 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:57.026228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:57.026781 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:57.026798 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:57.027309 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:57.027312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:57.027395 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:57.027880 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:57.027899 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:57.029680 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
W20260929 18:42:57.029992 16994 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:57.024667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:43997 (local address 127.7.157.126:39795)
0929 18:42:57.024810 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:57.024817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:57.024848 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:42:57.025002 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:57.025005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:57.025056 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:57.025122 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:57.025583 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:57.026115 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:57.026907 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:57.027200 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:57.027300 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:57.029874 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:43997: BlockingRecv error: recv got EOF from 127.7.157.67:43997 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 18:42:57 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:57 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:57.664157 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:57.659210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:37189)
0929 18:42:57.659462 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:57.659476 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:57.659586 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:57.659847 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:57.659849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:57.660016 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:57.660108 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:57.660113 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:57.660833 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:57.660835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:57.661374 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:57.661380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:57.661839 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:57.661841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:57.661902 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:57.662362 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:57.662372 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:57.664039 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":69,"threads_started":1}
W20260929 18:42:57.664319 16998 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:57.659229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:37189 (local address 127.7.157.126:39795)
0929 18:42:57.659478 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:57.659482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:57.659527 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:42:57.659675 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:57.659678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:57.659731 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:57.659823 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:57.660194 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:57.660748 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:57.661470 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:57.661735 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:57.661812 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:57.664190 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:37189: BlockingRecv error: recv got EOF from 127.7.157.65:37189 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
Sep 29 18:42:57 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:57 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:57.847554 16999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:57.842039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:57917)
0929 18:42:57.842379 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:57.842397 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:57.842540 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:57.842839 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:57.842842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:57.843057 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:57.843194 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:57.843201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:57.843934 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:57.843936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:57.844495 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:57.844502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:57.844946 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:57.844949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:57.845011 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:57.845476 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:57.845487 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:57.847399 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":78,"threads_started":1}
W20260929 18:42:57.847640 16998 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:57.842133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:57917 (local address 127.7.157.126:39795)
0929 18:42:57.842271 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:57.842276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:57.842491 (+ 215us) server_negotiation.cc:407] Connection header received
0929 18:42:57.842660 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:57.842663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:57.842718 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:57.842809 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:57.843295 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:57.843847 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:57.844614 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:57.844845 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:57.844934 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:57.847540 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:57917: BlockingRecv error: recv got EOF from 127.7.157.66:57917 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 18:42:58 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:58 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:58.035858 17000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:58.030647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:34775)
0929 18:42:58.030939 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:58.030953 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:58.031089 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:58.031383 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:58.031387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:58.031595 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:58.031693 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:58.031698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:58.032472 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:58.032474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:58.033020 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:58.033026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:58.033423 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:58.033426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:58.033482 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:58.033939 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:58.033949 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:58.035730 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
W20260929 18:42:58.035983 16998 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:58.030732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:34775 (local address 127.7.157.126:39795)
0929 18:42:58.030870 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:58.030875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:58.031024 (+ 149us) server_negotiation.cc:407] Connection header received
0929 18:42:58.031201 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:58.031205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:58.031262 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:58.031335 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:58.031829 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:58.032369 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:58.033120 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:58.033339 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:58.033416 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:58.035865 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:34775: BlockingRecv error: recv got EOF from 127.7.157.67:34775 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 18:42:58 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:58 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:58.670253 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:58.664919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:51775)
0929 18:42:58.665219 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:58.665232 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:58.665322 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:58.665609 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:58.665612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:58.665834 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:58.665931 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:58.665937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:58.666692 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:58.666695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:58.667291 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:58.667300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:58.667748 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:58.667751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:58.667835 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:58.668317 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:58.668328 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:58.670129 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":76,"threads_started":1}
W20260929 18:42:58.670526 17002 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:58.665057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:51775 (local address 127.7.157.126:39795)
0929 18:42:58.665263 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:58.665266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:58.665279 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:42:58.665437 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:58.665440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:58.665489 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:58.665572 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:58.666032 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:58.666568 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:58.667391 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:58.667616 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:58.667701 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:58.670275 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:51775: BlockingRecv error: recv got EOF from 127.7.157.65:51775 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":54,"threads_started":1}
Sep 29 18:42:58 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:58 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:58.853902 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:58.848478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:37377)
0929 18:42:58.848824 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:58.848840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:58.848952 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:58.849205 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:58.849208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:58.849400 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:58.849551 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:58.849559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:58.850332 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:58.850336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:58.851001 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:58.851011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:58.851507 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:58.851511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:58.851585 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:58.852020 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:58.852031 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:58.853766 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":87,"threads_started":1}
W20260929 18:42:58.854012 17002 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:58.848845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:37377 (local address 127.7.157.126:39795)
0929 18:42:58.848985 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:58.848990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:58.849005 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:42:58.849048 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:58.849051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:58.849106 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:58.849172 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:58.849700 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:58.850228 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:58.851146 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:58.851401 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:58.851506 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:58.853901 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:37377: BlockingRecv error: recv got EOF from 127.7.157.66:37377 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 18:42:59 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:59 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:59.041646 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:59.036637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:33767)
0929 18:42:59.036914 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:59.036926 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:59.037021 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:59.037225 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:59.037226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:59.037405 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:59.037494 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:59.037499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:59.038245 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:59.038248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:59.038808 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:59.038816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:59.039246 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:59.039249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:59.039313 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:59.039792 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:59.039820 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:59.041527 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":77,"threads_started":1}
W20260929 18:42:59.041827 17002 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:59.036667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:33767 (local address 127.7.157.126:39795)
0929 18:42:59.036797 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:59.036802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:59.037011 (+ 209us) server_negotiation.cc:407] Connection header received
0929 18:42:59.037091 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:59.037094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:59.037145 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:59.037221 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:59.037588 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:59.038148 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:59.038918 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:59.039164 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:59.039240 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:59.041702 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:33767: BlockingRecv error: recv got EOF from 127.7.157.67:33767 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 18:42:59 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:59 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:59.675858 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:59.670972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:48793)
0929 18:42:59.671233 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:59.671249 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:59.671343 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:59.671555 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:59.671557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:59.671723 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:59.671834 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:59.671839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:59.672538 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:59.672539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:59.673094 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:59.673100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:59.673526 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:59.673529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:59.673587 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:59.674066 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:59.674078 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:59.675726 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":109,"threads_started":1}
W20260929 18:42:59.676009 17006 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:59.671134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:48793 (local address 127.7.157.126:39795)
0929 18:42:59.671353 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:59.671356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:59.671389 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:42:59.671423 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:59.671425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:59.671467 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:59.671531 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:59.671933 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:59.672463 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:59.673184 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:59.673441 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:59.673530 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:59.675882 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:48793: BlockingRecv error: recv got EOF from 127.7.157.65:48793 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":70,"threads_started":1}
Sep 29 18:42:59 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:42:59 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:42:59.859879 17007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:42:59.854565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:37683)
0929 18:42:59.854827 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 18:42:59.854841 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:42:59.854957 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:42:59.855214 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:42:59.855217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:42:59.855405 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:42:59.855494 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:59.855499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:59.856401 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:59.856405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:59.856956 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:42:59.856964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:42:59.857380 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:42:59.857383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:42:59.857461 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:59.857948 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:42:59.857959 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:42:59.859729 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":75,"threads_started":1}
W20260929 18:42:59.859970 17006 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:42:59.854642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:37683 (local address 127.7.157.126:39795)
0929 18:42:59.854782 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:42:59.854786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:42:59.854918 (+ 132us) server_negotiation.cc:407] Connection header received
0929 18:42:59.855042 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:42:59.855045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:42:59.855108 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:42:59.855172 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:42:59.855595 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:59.856292 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:59.857059 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:42:59.857280 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:42:59.857363 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:42:59.859873 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:37683: BlockingRecv error: recv got EOF from 127.7.157.66:37683 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 18:43:00 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:00 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:00.047719 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:00.042463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:57949)
0929 18:43:00.042788 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:00.042802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:00.042911 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:00.043170 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:00.043172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:00.043358 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:00.043459 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:00.043465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:00.044174 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:00.044177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:00.044775 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:00.044784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:00.045178 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:00.045181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:00.045249 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:00.045714 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:00.045726 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:00.047584 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":106,"threads_started":1}
W20260929 18:43:00.047811 17006 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:00.042566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:57949 (local address 127.7.157.126:39795)
0929 18:43:00.042713 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:00.042717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:00.042868 (+ 151us) server_negotiation.cc:407] Connection header received
0929 18:43:00.042988 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:00.042990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:00.043039 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:00.043142 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:00.043557 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:00.044069 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:00.044871 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:00.045085 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:00.045151 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:00.047707 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:57949: BlockingRecv error: recv got EOF from 127.7.157.67:57949 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 18:43:00 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:00 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:00.689035 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:00.676626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:53501)
0929 18:43:00.676907 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:00.676920 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:00.677010 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:00.677274 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:00.677277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:00.677510 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:00.677613 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:00.677620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:00.678456 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:00.678460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:00.679060 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:00.679092 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:00.679528 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:00.679532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:00.679616 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:00.686929 (+ 7313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:00.686942 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:00.688840 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":92,"threads_started":1}
W20260929 18:43:00.689162 17010 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:00.676757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:53501 (local address 127.7.157.126:39795)
0929 18:43:00.676971 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:00.676975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:00.676988 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:43:00.677092 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:00.677094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:00.677148 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:00.677226 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:00.677720 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:00.678346 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:00.679200 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:00.679425 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:00.679510 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:00.689026 (+ 9516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:53501: BlockingRecv error: recv got EOF from 127.7.157.65:53501 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":60,"threads_started":1}
Sep 29 18:43:00 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:00 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:00.866284 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:00.860754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:55769)
0929 18:43:00.861040 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:00.861055 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:00.861167 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:00.861422 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:00.861425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:00.861622 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:00.861712 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:00.861718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:00.862477 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:00.862481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:00.863060 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:00.863105 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:00.863568 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:00.863572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:00.863655 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:00.864193 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:00.864208 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:00.866141 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":103,"threads_started":1}
W20260929 18:43:00.866377 17010 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:00.860860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:55769 (local address 127.7.157.126:39795)
0929 18:43:00.861003 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:00.861009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:00.861126 (+ 117us) server_negotiation.cc:407] Connection header received
0929 18:43:00.861257 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:00.861260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:00.861315 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:00.861383 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:00.861833 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:00.862375 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:00.863211 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:00.863442 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:00.863532 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:00.866271 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:55769: BlockingRecv error: recv got EOF from 127.7.157.66:55769 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 18:43:01 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:01 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:01.054352 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:01.048697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:48175)
0929 18:43:01.048994 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:01.049010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:01.049145 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:01.049408 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:01.049411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:01.049611 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:01.049727 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:01.049734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:01.050596 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:01.050600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:01.051387 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:01.051397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:01.051824 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:01.051827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:01.051912 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:01.052390 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:01.052402 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:01.054198 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":104,"threads_started":1}
W20260929 18:43:01.054446 17010 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:01.048825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:48175 (local address 127.7.157.126:39795)
0929 18:43:01.048996 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:01.049001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:01.049081 (+ 80us) server_negotiation.cc:407] Connection header received
0929 18:43:01.049237 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:01.049240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:01.049290 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:01.049372 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:01.049845 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:01.050471 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:01.051493 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:01.051721 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:01.051802 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:01.054349 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:48175: BlockingRecv error: recv got EOF from 127.7.157.67:48175 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 18:43:01 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:01 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:01.695252 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:01.689919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:34843)
0929 18:43:01.690189 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:01.690203 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:01.690298 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:01.690573 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:01.690576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:01.690791 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:01.690911 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:01.690919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:01.691832 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:01.691835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:01.692394 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:01.692402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:01.692778 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:01.692781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:01.692840 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:01.693289 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:01.693300 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:01.695134 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
W20260929 18:43:01.695370 17014 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:01.690020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:34843 (local address 127.7.157.126:39795)
0929 18:43:01.690295 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:01.690299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:01.690315 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:43:01.690407 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:01.690409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:01.690456 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:01.690544 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:01.691029 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:01.691734 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:01.692483 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:01.692703 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:01.692767 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:01.695258 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:34843: BlockingRecv error: recv got EOF from 127.7.157.65:34843 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":71,"threads_started":1}
Sep 29 18:43:01 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:01 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:01.872113 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:01.867006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:50847)
0929 18:43:01.867316 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:01.867330 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:01.867423 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:01.867630 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:01.867633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:01.867806 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:01.867899 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:01.867904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:01.868597 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:01.868599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:01.869164 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:01.869172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:01.869602 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:01.869606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:01.869676 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:01.870134 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:01.870146 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:01.871958 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":99,"threads_started":1}
W20260929 18:43:01.872207 17014 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:01.867118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:50847 (local address 127.7.157.126:39795)
0929 18:43:01.867268 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:01.867272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:01.867396 (+ 124us) server_negotiation.cc:407] Connection header received
0929 18:43:01.867497 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:01.867499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:01.867541 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:01.867597 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:01.867984 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:01.868503 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:01.869255 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:01.869480 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:01.869553 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:01.872106 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:50847: BlockingRecv error: recv got EOF from 127.7.157.66:50847 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 18:43:02 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:02 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:02.060474 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:02.055359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:37557)
0929 18:43:02.055615 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:02.055631 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:02.055727 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:02.055939 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:02.055941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:02.056111 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:02.056216 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:02.056221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:02.056941 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:02.056944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:02.057525 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:02.057534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:02.057905 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:02.057907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:02.057967 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:02.058466 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:02.058478 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:02.060322 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
W20260929 18:43:02.060680 17014 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:02.055515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:37557 (local address 127.7.157.126:39795)
0929 18:43:02.055633 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:02.055636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:02.055687 (+ 51us) server_negotiation.cc:407] Connection header received
0929 18:43:02.055806 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:02.055809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:02.055855 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:02.055914 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:02.056302 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:02.056821 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:02.057617 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:02.057824 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:02.057888 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:02.060483 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:37557: BlockingRecv error: recv got EOF from 127.7.157.67:37557 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 18:43:02 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:02 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:02.700887 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:02.696022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:56833)
0929 18:43:02.696293 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:02.696307 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:02.696408 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:02.696631 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:02.696633 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:02.696804 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:02.696897 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:02.696902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:02.697603 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:02.697605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:02.698158 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:02.698165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:02.698545 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:02.698548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:02.698611 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:02.699053 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:02.699090 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:02.700751 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
W20260929 18:43:02.701000 17018 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:02.696119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:56833 (local address 127.7.157.126:39795)
0929 18:43:02.696370 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:02.696374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:02.696388 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:43:02.696486 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:02.696488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:02.696531 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:02.696606 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:02.696984 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:02.697524 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:02.698243 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:02.698463 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:02.698532 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:02.700887 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:56833: BlockingRecv error: recv got EOF from 127.7.157.65:56833 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
Sep 29 18:43:02 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:02 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:02.877835 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:02.872772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:42469)
0929 18:43:02.873033 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:02.873046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:02.873164 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:02.873386 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:02.873388 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:02.873573 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:02.873711 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:02.873718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:02.874453 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:02.874456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:02.875058 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:02.875104 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:02.875517 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:02.875520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:02.875578 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:02.876041 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:02.876053 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:02.877710 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":66,"threads_started":1}
W20260929 18:43:02.878079 17018 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:02.872865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:42469 (local address 127.7.157.126:39795)
0929 18:43:02.873011 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:02.873014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:02.873116 (+ 102us) server_negotiation.cc:407] Connection header received
0929 18:43:02.873245 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:02.873247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:02.873294 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:02.873362 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:02.873800 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:02.874356 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:02.875201 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:02.875430 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:02.875502 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:02.877860 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:42469: BlockingRecv error: recv got EOF from 127.7.157.66:42469 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 18:43:03 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:03 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:03.066578 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:03.061203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:34041)
0929 18:43:03.061469 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:03.061482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:03.061578 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:03.061824 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:03.061826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:03.061993 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:03.062088 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:03.062093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:03.062850 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:03.062853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:03.063457 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:03.063467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:03.063876 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:03.063879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:03.063934 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:03.064416 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:03.064428 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:03.066460 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":84,"threads_started":1}
W20260929 18:43:03.066651 17018 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:03.061286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:34041 (local address 127.7.157.126:39795)
0929 18:43:03.061434 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:03.061437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:03.061570 (+ 133us) server_negotiation.cc:407] Connection header received
0929 18:43:03.061675 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:03.061677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:03.061727 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:03.061800 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:03.062222 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:03.062763 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:03.063545 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:03.063793 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:03.063867 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:03.066560 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:34041: BlockingRecv error: recv got EOF from 127.7.157.67:34041 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 18:43:03 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:03 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:03.706985 17021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:03.701825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:51993)
0929 18:43:03.702093 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:03.702107 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:03.702201 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:03.702447 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:03.702449 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:03.702627 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:03.702719 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:03.702724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:03.703527 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:03.703529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:03.704075 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:03.704083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:03.704475 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:03.704477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:03.704538 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:03.705011 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:03.705022 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:03.706830 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
W20260929 18:43:03.707161 17022 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:03.701963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:51993 (local address 127.7.157.126:39795)
0929 18:43:03.702171 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:03.702175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:03.702201 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:43:03.702287 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:03.702289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:03.702333 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:03.702421 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:03.702840 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:03.703448 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:03.704162 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:03.704389 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:03.704459 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:03.707005 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:51993: BlockingRecv error: recv got EOF from 127.7.157.65:51993 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":59,"threads_started":1}
Sep 29 18:43:03 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:03 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:03.883855 17023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:03.878576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:34163)
0929 18:43:03.878870 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:03.878884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:03.878985 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:03.879232 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:03.879235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:03.879467 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:03.879564 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:03.879570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:03.880309 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:03.880312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:03.880861 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:03.880869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:03.881289 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:03.881293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:03.881372 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:03.881841 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:03.881855 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:03.883719 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":102,"threads_started":1}
W20260929 18:43:03.883970 17022 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:03.878667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:34163 (local address 127.7.157.126:39795)
0929 18:43:03.878832 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:03.878838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:03.878948 (+ 110us) server_negotiation.cc:407] Connection header received
0929 18:43:03.879079 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:03.879082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:03.879129 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:03.879191 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:03.879673 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:03.880195 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:03.880968 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:03.881186 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:03.881268 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:03.883869 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:34163: BlockingRecv error: recv got EOF from 127.7.157.66:34163 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 18:43:04 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:04 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:04.077785 17024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:04.067375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:55507)
0929 18:43:04.067694 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:04.067710 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:04.067821 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:04.068044 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:04.068046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:04.068238 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:04.068339 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:04.068346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:04.069134 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:04.069137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:04.069765 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:04.069776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:04.070256 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:04.070260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:04.070335 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:04.075496 (+ 5161us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:04.075510 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:04.077625 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":67,"threads_started":1}
W20260929 18:43:04.077960 17022 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:04.067531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:55507 (local address 127.7.157.126:39795)
0929 18:43:04.067702 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:04.067705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:04.067776 (+ 71us) server_negotiation.cc:407] Connection header received
0929 18:43:04.067909 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:04.067912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:04.067956 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:04.068012 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:04.068452 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:04.069017 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:04.069892 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:04.070126 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:04.070209 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:04.077823 (+ 7614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:55507: BlockingRecv error: recv got EOF from 127.7.157.67:55507 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 18:43:04 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:04 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:04.712898 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:04.707780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:44187)
0929 18:43:04.708040 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:04.708053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:04.708150 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:04.708453 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:04.708455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:04.708628 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:04.708725 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:04.708730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:04.709431 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:04.709433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:04.709981 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:04.709988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:04.710396 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:04.710399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:04.710456 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:04.710907 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:04.710917 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:04.712760 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
W20260929 18:43:04.713035 17026 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:04.707828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:44187 (local address 127.7.157.126:39795)
0929 18:43:04.708041 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:04.708045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:04.708117 (+ 72us) server_negotiation.cc:407] Connection header received
0929 18:43:04.708269 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:04.708272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:04.708321 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:04.708419 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:04.708816 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:04.709350 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:04.710077 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:04.710313 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:04.710388 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:04.712903 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:44187: BlockingRecv error: recv got EOF from 127.7.157.65:44187 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":53,"threads_started":1}
Sep 29 18:43:04 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:04 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:04.890146 17027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:04.884602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:40809)
0929 18:43:04.884874 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:04.884890 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:04.884999 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:04.885291 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:04.885294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:04.885490 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:04.885581 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:04.885587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:04.886464 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:04.886467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:04.887046 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:04.887055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:04.887552 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:04.887554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:04.887615 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:04.888120 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:04.888132 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:04.890015 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":64,"threads_started":1}
W20260929 18:43:04.890298 17026 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:04.884816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:40809 (local address 127.7.157.126:39795)
0929 18:43:04.884957 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:04.884961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:04.884974 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:43:04.885124 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:04.885126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:04.885174 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:04.885248 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:04.885804 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:04.886339 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:04.887194 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:04.887457 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:04.887540 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:04.890162 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:40809: BlockingRecv error: recv got EOF from 127.7.157.66:40809 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 18:43:05 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:05 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:05.083811 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:05.078536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:46791)
0929 18:43:05.078824 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:05.078859 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:05.078970 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:05.079246 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:05.079249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:05.079448 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:05.079537 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:05.079542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:05.080269 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:05.080273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:05.080915 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:05.080924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:05.081334 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:05.081338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:05.081417 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:05.081893 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:05.081904 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:05.083670 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":79,"threads_started":1}
W20260929 18:43:05.083971 17026 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:05.078637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:46791 (local address 127.7.157.126:39795)
0929 18:43:05.078797 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:05.078802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:05.078928 (+ 126us) server_negotiation.cc:407] Connection header received
0929 18:43:05.079058 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:05.079062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:05.079138 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:05.079206 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:05.079649 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:05.080165 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:05.081017 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:05.081235 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:05.081320 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:05.083842 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:46791: BlockingRecv error: recv got EOF from 127.7.157.67:46791 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 18:43:05 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:05 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:05.724860 17029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:05.713716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:55119)
0929 18:43:05.714078 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:05.714093 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:05.714186 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:05.714462 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:05.714472 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:05.714691 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:05.714787 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:05.714793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:05.715681 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:05.715684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:05.716263 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:05.716272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:05.716706 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:05.716709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:05.716771 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:05.722863 (+ 6092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:05.722877 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:05.724692 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":90,"threads_started":1}
W20260929 18:43:05.724979 17030 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:05.713819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:55119 (local address 127.7.157.126:39795)
0929 18:43:05.714112 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:05.714117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:05.714162 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:43:05.714277 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:05.714279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:05.714335 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:05.714416 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:05.714879 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:05.715534 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:05.716362 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:05.716590 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:05.716662 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:05.724862 (+ 8200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:55119: BlockingRecv error: recv got EOF from 127.7.157.65:55119 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":74,"threads_started":1}
Sep 29 18:43:05 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:05 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:05.895973 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:05.890882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:36047)
0929 18:43:05.891180 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:05.891192 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:05.891289 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:05.891498 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:05.891501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:05.891670 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:05.891764 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:05.891769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:05.892451 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:05.892454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:05.893026 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:05.893033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:05.893490 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:05.893492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:05.893549 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:05.894081 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:05.894092 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:05.895845 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":81,"threads_started":1}
W20260929 18:43:05.896111 17030 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:05.890968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:36047 (local address 127.7.157.126:39795)
0929 18:43:05.891115 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:05.891120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:05.891253 (+ 133us) server_negotiation.cc:407] Connection header received
0929 18:43:05.891367 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:05.891369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:05.891415 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:05.891473 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:05.891847 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:05.892365 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:05.893153 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:05.893396 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:05.893483 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:05.895997 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:36047: BlockingRecv error: recv got EOF from 127.7.157.66:36047 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 18:43:06 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:06 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.67@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:06.089588 17032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:06.084534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.67:56923)
0929 18:43:06.084778 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:06.084790 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:06.084891 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:06.085100 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:06.085101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:06.085267 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:06.085366 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:06.085371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:06.086044 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:06.086046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:06.086617 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:06.086624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:06.087058 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:06.087061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:06.087162 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:06.087617 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:06.087628 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:06.089410 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":111,"threads_started":1}
W20260929 18:43:06.089781 17030 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:06.084588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:56923 (local address 127.7.157.126:39795)
0929 18:43:06.084725 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:06.084730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:06.084856 (+ 126us) server_negotiation.cc:407] Connection header received
0929 18:43:06.084962 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:06.084964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:06.085009 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:06.085073 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:06.085448 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:06.085964 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:06.086734 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:06.086960 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:06.087042 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:06.089554 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.67:56923: BlockingRecv error: recv got EOF from 127.7.157.67:56923 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 18:43:06 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:06 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.65@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:06.731149 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:06.725822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.65:42931)
0929 18:43:06.726099 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:06.726114 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:06.726209 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:06.726482 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:06.726486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:06.726667 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:06.726807 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:06.726813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:06.727634 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:06.727637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:06.728397 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:06.728408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:06.728846 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:06.728849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:06.728931 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:06.729362 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:06.729373 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:06.731009 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":78,"threads_started":1}
W20260929 18:43:06.731272 17034 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:06.725985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:42931 (local address 127.7.157.126:39795)
0929 18:43:06.726205 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:06.726209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:06.726230 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:43:06.726281 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:06.726283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:06.726324 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:06.726405 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:06.726914 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:06.727529 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:06.728516 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:06.728743 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:06.728827 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:06.731162 (+ 2335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.65:42931: BlockingRecv error: recv got EOF from 127.7.157.65:42931 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
Sep 29 18:43:06 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:43:06 dist-test-slave-smpn krb5kdc[15976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.157.66@KRBTEST.COM for oryx/127.7.157.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:43:06.901772 17035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:06.896895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39795 (local address 127.7.157.66:39929)
0929 18:43:06.897144 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:06.897158 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:06.897264 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:06.897480 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:06.897482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:06.897653 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:06.897750 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:06.897755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:06.898433 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:06.898435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:06.899003 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:06.899011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:06.899423 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:06.899425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:06.899484 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:06.899929 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:06.899940 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:06.901633 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.157.126:39795: Server oryx/127.7.157.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":67,"threads_started":1}
W20260929 18:43:06.902033 17034 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:06.896981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:39929 (local address 127.7.157.126:39795)
0929 18:43:06.897104 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:06.897108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:06.897225 (+ 117us) server_negotiation.cc:407] Connection header received
0929 18:43:06.897337 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:06.897339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:06.897385 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:06.897457 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:06.897833 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:06.898353 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:06.899115 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:06.899337 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:06.899411 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:06.901790 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.157.66:39929: BlockingRecv error: recv got EOF from 127.7.157.66:39929 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:43:06.982306 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 16559
I20260929 18:43:06.987092 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 16711
I20260929 18:43:06.991608 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 16843
I20260929 18:43:06.996112 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 16489
2026-09-29T18:43:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17857 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T18:43:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:43:07Z Disabled control of system clock
I20260929 18:43:07.023350 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:35461
--webserver_interface=127.7.157.126
--webserver_port=0
--builtin_ntp_servers=127.7.157.84:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:35461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 18:43:07.131973 17039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:43:07.132251 17039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:43:07.132285 17039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:43:07.136348 17039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:43:07.136413 17039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:43:07.136428 17039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:43:07.136443 17039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:43:07.136457 17039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:43:07.141386 17039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:35461
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:35461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:43:07.142482 17039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:43:07.143851 17039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:43:07.152539 17039 server_base.cc:1034] running on GCE node
W20260929 18:43:07.152478 17047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:07.152478 17044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:07.152493 17045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:07.153297 17039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:43:07.154505 17039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:43:07.155670 17039 hybrid_clock.cc:648] HybridClock initialized: now 1790707387155641 us; error 46 us; skew 500 ppm
I20260929 18:43:07.157552 17039 webserver.cc:460] Webserver started at http://127.7.157.126:45385/ using document root <none> and password file <none>
I20260929 18:43:07.158171 17039 fs_manager.cc:359] Metadata directory not provided
I20260929 18:43:07.158239 17039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:43:07.158434 17039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:43:07.160521 17039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "247424bd880d499e911928743176b755"
format_stamp: "Formatted at 2026-09-29 18:43:07 on dist-test-slave-smpn"
server_key: "fb42adbce1aa073e651a731c53e16889"
server_key_iv: "87ff2f99bf4018e4d8580b297d5000af"
server_key_version: "encryptionkey@0"
I20260929 18:43:07.161072 17039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "247424bd880d499e911928743176b755"
format_stamp: "Formatted at 2026-09-29 18:43:07 on dist-test-slave-smpn"
server_key: "fb42adbce1aa073e651a731c53e16889"
server_key_iv: "87ff2f99bf4018e4d8580b297d5000af"
server_key_version: "encryptionkey@0"
I20260929 18:43:07.164705 17039 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:43:07.167016 17053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:07.168243 17039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:43:07.168371 17039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "247424bd880d499e911928743176b755"
format_stamp: "Formatted at 2026-09-29 18:43:07 on dist-test-slave-smpn"
server_key: "fb42adbce1aa073e651a731c53e16889"
server_key_iv: "87ff2f99bf4018e4d8580b297d5000af"
server_key_version: "encryptionkey@0"
I20260929 18:43:07.168473 17039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:43:07.184792 17039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:43:07.185474 17039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:43:07.185645 17039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:43:07.192365 17039 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:35461
I20260929 18:43:07.192377 17105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:35461 every 8 connection(s)
I20260929 18:43:07.193481 17039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:43:07.196749 17106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:43:07.198753 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 17039
I20260929 18:43:07.198861 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:43:07.199144 7797 external_mini_cluster.cc:1428] Setting key d1688796cb802d144f30593679cb42a3
I20260929 18:43:07.202613 17106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755: Bootstrap starting.
I20260929 18:43:07.204783 17106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755: Neither blocks nor log segments found. Creating new log.
I20260929 18:43:07.205474 17106 log.cc:824] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755: Log is configured to *not* fsync() on all Append() calls
I20260929 18:43:07.207588 17106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755: No bootstrap required, opened a new log
I20260929 18:43:07.208133 17109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:07.200524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52352 (local address 127.7.157.126:35461)
0929 18:43:07.201166 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:07.201185 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:07.201221 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:43:07.201849 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:07.201866 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:07.202265 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:07.202616 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:07.203040 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:07.204048 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:07.204875 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:07.205350 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:07.205491 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:07.206327 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:07.206333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:07.206353 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:43:07.206719 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:07.206772 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:43:07.206790 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:07.206869 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:07.207030 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 18:43:07.207321 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":123,"threads_started":1}
I20260929 18:43:07.211591 17106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247424bd880d499e911928743176b755" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35461 } }
I20260929 18:43:07.211836 17106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:43:07.211895 17106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247424bd880d499e911928743176b755, State: Initialized, Role: FOLLOWER
I20260929 18:43:07.212314 17106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247424bd880d499e911928743176b755" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35461 } }
I20260929 18:43:07.212453 17106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:43:07.212522 17106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:43:07.212635 17106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:43:07.213580 17106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247424bd880d499e911928743176b755" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35461 } }
I20260929 18:43:07.213918 17106 leader_election.cc:302] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 247424bd880d499e911928743176b755; no voters:
I20260929 18:43:07.214226 17106 leader_election.cc:288] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:43:07.214313 17111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:43:07.214533 17111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [term 1 LEADER]: Becoming Leader. State: Replica: 247424bd880d499e911928743176b755, State: Running, Role: LEADER
I20260929 18:43:07.215209 17111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247424bd880d499e911928743176b755" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35461 } }
I20260929 18:43:07.215544 17106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:43:07.216737 17113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 247424bd880d499e911928743176b755. Latest consensus state: current_term: 1 leader_uuid: "247424bd880d499e911928743176b755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247424bd880d499e911928743176b755" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35461 } } }
I20260929 18:43:07.216840 17113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [sys.catalog]: This master's current role is: LEADER
I20260929 18:43:07.217167 17112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "247424bd880d499e911928743176b755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247424bd880d499e911928743176b755" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35461 } } }
I20260929 18:43:07.217257 17112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755 [sys.catalog]: This master's current role is: LEADER
I20260929 18:43:07.217273 17122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:43:07.220454 17122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:43:07.225751 17122 catalog_manager.cc:1347] Generated new cluster ID: 17a0623de6864c8d9b66f81bd6686466
I20260929 18:43:07.225829 17122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:43:07.239873 17122 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:43:07.241091 17122 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:43:07.248083 17122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 247424bd880d499e911928743176b755: Generated new TSK 0
I20260929 18:43:07.248978 17122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:43:07.254153 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35461
--builtin_ntp_servers=127.7.157.84:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 18:43:07.361725 17130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:43:07.361968 17130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:43:07.362004 17130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:43:07.366042 17130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:43:07.366107 17130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:43:07.366163 17130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:43:07.370993 17130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:43:07.372412 17130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:43:07.373895 17130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:43:07.383320 17135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:07.383320 17138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:07.383417 17130 server_base.cc:1034] running on GCE node
W20260929 18:43:07.383325 17136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:07.384039 17130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:43:07.384672 17130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:43:07.385831 17130 hybrid_clock.cc:648] HybridClock initialized: now 1790707387385804 us; error 35 us; skew 500 ppm
I20260929 18:43:07.387576 17130 webserver.cc:460] Webserver started at http://127.7.157.65:43535/ using document root <none> and password file <none>
I20260929 18:43:07.388120 17130 fs_manager.cc:359] Metadata directory not provided
I20260929 18:43:07.388191 17130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:43:07.388386 17130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:43:07.390435 17130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "296b943f0e494e7a94c73d1e4fcc1381"
format_stamp: "Formatted at 2026-09-29 18:43:07 on dist-test-slave-smpn"
server_key: "1252a48e460fb79bad9acccb33ad4bce"
server_key_iv: "92d61ea350e1433a50c8781b5dd88e3f"
server_key_version: "encryptionkey@0"
I20260929 18:43:07.390969 17130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "296b943f0e494e7a94c73d1e4fcc1381"
format_stamp: "Formatted at 2026-09-29 18:43:07 on dist-test-slave-smpn"
server_key: "1252a48e460fb79bad9acccb33ad4bce"
server_key_iv: "92d61ea350e1433a50c8781b5dd88e3f"
server_key_version: "encryptionkey@0"
I20260929 18:43:07.394548 17130 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260929 18:43:07.397071 17144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:07.398321 17130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:43:07.398448 17130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "296b943f0e494e7a94c73d1e4fcc1381"
format_stamp: "Formatted at 2026-09-29 18:43:07 on dist-test-slave-smpn"
server_key: "1252a48e460fb79bad9acccb33ad4bce"
server_key_iv: "92d61ea350e1433a50c8781b5dd88e3f"
server_key_version: "encryptionkey@0"
I20260929 18:43:07.398558 17130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:43:07.414465 17130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:43:07.415166 17130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:43:07.415341 17130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:43:07.415936 17130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:43:07.416915 17130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:43:07.416966 17130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:07.417021 17130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:43:07.417050 17130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:07.425791 17130 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:39317
I20260929 18:43:07.425809 17257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:39317 every 8 connection(s)
I20260929 18:43:07.426730 17130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:43:07.430413 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 17130
I20260929 18:43:07.430501 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:43:07.430738 7797 external_mini_cluster.cc:1428] Setting key 38788ea46c259db187b0e6e1198761e4
I20260929 18:43:07.432466 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35461
--builtin_ntp_servers=127.7.157.84:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 18:43:07.436002 17109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:07.428506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:46169 (local address 127.7.157.126:35461)
0929 18:43:07.428702 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:07.428706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:07.429777 (+ 1071us) server_negotiation.cc:407] Connection header received
0929 18:43:07.431256 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:07.431260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:07.431320 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:07.431405 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:07.432302 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:07.432888 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:07.434039 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:07.434268 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:07.434369 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:07.435483 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:07.435486 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:07.435489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:43:07.435501 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:07.435530 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:43:07.435539 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:07.435606 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:07.435776 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 18:43:07.435844 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 18:43:07.436812 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:07.428904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:35461 (local address 127.7.157.65:46169)
0929 18:43:07.429598 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:07.429650 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:07.431009 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:07.431582 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:07.431589 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:07.431675 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:07.432164 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:07.432177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:07.433017 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:07.433019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:07.433916 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:07.433926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:07.434420 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:07.434424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:07.434604 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:07.435226 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:43:07.435227 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:07.435267 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:43:07.435331 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:43:07.435616 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:43:07.435625 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:43:07.435632 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:43:07.435755 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 18:43:07.436037 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":109,"threads_started":1}
I20260929 18:43:07.438060 17258 heartbeater.cc:347] Connected to a master server at 127.7.157.126:35461
I20260929 18:43:07.438309 17258 heartbeater.cc:464] Registering TS with master...
I20260929 18:43:07.438930 17258 heartbeater.cc:511] Master 127.7.157.126:35461 requested a full tablet report, sending...
I20260929 18:43:07.440696 17070 ts_manager.cc:194] Registered new tserver with Master: 296b943f0e494e7a94c73d1e4fcc1381 (127.7.157.65:39317)
I20260929 18:43:07.442274 17070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.157.65:46169
W20260929 18:43:07.540534 17261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:43:07.540802 17261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:43:07.540848 17261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:43:07.544807 17261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:43:07.544872 17261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:43:07.544925 17261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:43:07.549701 17261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:43:07.550766 17261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:43:07.552152 17261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:43:07.561529 17267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:07.561529 17266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:07.561524 17269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:07.561750 17261 server_base.cc:1034] running on GCE node
I20260929 18:43:07.562168 17261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:43:07.562795 17261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:43:07.563951 17261 hybrid_clock.cc:648] HybridClock initialized: now 1790707387563933 us; error 28 us; skew 500 ppm
I20260929 18:43:07.565647 17261 webserver.cc:460] Webserver started at http://127.7.157.66:34635/ using document root <none> and password file <none>
I20260929 18:43:07.566186 17261 fs_manager.cc:359] Metadata directory not provided
I20260929 18:43:07.566250 17261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:43:07.566430 17261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:43:07.568463 17261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "d3d2cd5e0a1c4874840f9245f96dd4d7"
format_stamp: "Formatted at 2026-09-29 18:43:07 on dist-test-slave-smpn"
server_key: "5d3753e86a98f7f8b869ad9b0ef5a91e"
server_key_iv: "0f39e072d50200d41de918bacb546b3e"
server_key_version: "encryptionkey@0"
I20260929 18:43:07.568990 17261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "d3d2cd5e0a1c4874840f9245f96dd4d7"
format_stamp: "Formatted at 2026-09-29 18:43:07 on dist-test-slave-smpn"
server_key: "5d3753e86a98f7f8b869ad9b0ef5a91e"
server_key_iv: "0f39e072d50200d41de918bacb546b3e"
server_key_version: "encryptionkey@0"
I20260929 18:43:07.572690 17261 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260929 18:43:07.575223 17275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:07.576464 17261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260929 18:43:07.576572 17261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "d3d2cd5e0a1c4874840f9245f96dd4d7"
format_stamp: "Formatted at 2026-09-29 18:43:07 on dist-test-slave-smpn"
server_key: "5d3753e86a98f7f8b869ad9b0ef5a91e"
server_key_iv: "0f39e072d50200d41de918bacb546b3e"
server_key_version: "encryptionkey@0"
I20260929 18:43:07.576673 17261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:43:07.600594 17261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:43:07.601307 17261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:43:07.601511 17261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:43:07.602108 17261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:43:07.603102 17261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:43:07.603156 17261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:07.603210 17261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:43:07.603237 17261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:07.611325 17261 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:37529
I20260929 18:43:07.611356 17388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:37529 every 8 connection(s)
I20260929 18:43:07.612231 17261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:43:07.618432 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 17261
I20260929 18:43:07.618544 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:43:07.618865 7797 external_mini_cluster.cc:1428] Setting key 771d79c240b2ddd2924387b124df8334
I20260929 18:43:07.621029 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35461
--builtin_ntp_servers=127.7.157.84:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 18:43:07.621428 17109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:07.613966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:60019 (local address 127.7.157.126:35461)
0929 18:43:07.614104 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:07.614108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:07.615282 (+ 1174us) server_negotiation.cc:407] Connection header received
0929 18:43:07.616641 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:07.616644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:07.616691 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:07.616762 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:07.617635 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:07.618137 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:07.619582 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:07.619853 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:07.619953 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:07.620996 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:07.620999 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:07.621001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:43:07.621012 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:07.621033 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:43:07.621041 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:07.621102 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:07.621258 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 18:43:07.621316 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:43:07.622570 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:07.614441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:35461 (local address 127.7.157.66:60019)
0929 18:43:07.615136 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:07.615187 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:07.616459 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:07.616927 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:07.616933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:07.617029 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:07.617525 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:07.617537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:07.618237 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:07.618240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:07.619473 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:07.619484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:07.619986 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:07.619989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:07.620134 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:07.620713 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:43:07.620715 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:07.620768 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:43:07.620851 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:43:07.621117 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:43:07.621126 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:43:07.621132 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:43:07.621258 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 18:43:07.621555 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":110,"threads_started":1}
I20260929 18:43:07.623755 17389 heartbeater.cc:347] Connected to a master server at 127.7.157.126:35461
I20260929 18:43:07.623965 17389 heartbeater.cc:464] Registering TS with master...
I20260929 18:43:07.624465 17389 heartbeater.cc:511] Master 127.7.157.126:35461 requested a full tablet report, sending...
I20260929 18:43:07.625550 17070 ts_manager.cc:194] Registered new tserver with Master: d3d2cd5e0a1c4874840f9245f96dd4d7 (127.7.157.66:37529)
I20260929 18:43:07.626327 17070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.157.66:60019
W20260929 18:43:07.730557 17392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:43:07.730824 17392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:43:07.730867 17392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:43:07.734778 17392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:43:07.734843 17392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:43:07.734894 17392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:43:07.739524 17392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:43:07.740494 17392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:43:07.741770 17392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:43:07.750676 17400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:07.750689 17397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:07.750782 17392 server_base.cc:1034] running on GCE node
W20260929 18:43:07.750715 17398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:07.751363 17392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:43:07.751986 17392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:43:07.753144 17392 hybrid_clock.cc:648] HybridClock initialized: now 1790707387753123 us; error 30 us; skew 500 ppm
I20260929 18:43:07.754889 17392 webserver.cc:460] Webserver started at http://127.7.157.67:36431/ using document root <none> and password file <none>
I20260929 18:43:07.755481 17392 fs_manager.cc:359] Metadata directory not provided
I20260929 18:43:07.755548 17392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:43:07.755735 17392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:43:07.757938 17392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "6443a98fd82d49168d6651833e034818"
format_stamp: "Formatted at 2026-09-29 18:43:07 on dist-test-slave-smpn"
server_key: "befab23fcfb18db3220526ad4da9e50d"
server_key_iv: "dbb4a450a18efd260f2f50537caa0bfa"
server_key_version: "encryptionkey@0"
I20260929 18:43:07.758510 17392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "6443a98fd82d49168d6651833e034818"
format_stamp: "Formatted at 2026-09-29 18:43:07 on dist-test-slave-smpn"
server_key: "befab23fcfb18db3220526ad4da9e50d"
server_key_iv: "dbb4a450a18efd260f2f50537caa0bfa"
server_key_version: "encryptionkey@0"
I20260929 18:43:07.761994 17392 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:43:07.764400 17406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:07.765566 17392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:43:07.765666 17392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "6443a98fd82d49168d6651833e034818"
format_stamp: "Formatted at 2026-09-29 18:43:07 on dist-test-slave-smpn"
server_key: "befab23fcfb18db3220526ad4da9e50d"
server_key_iv: "dbb4a450a18efd260f2f50537caa0bfa"
server_key_version: "encryptionkey@0"
I20260929 18:43:07.765758 17392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:43:07.782521 17392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:43:07.783264 17392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:43:07.783432 17392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:43:07.784001 17392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:43:07.784936 17392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:43:07.784981 17392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:07.785025 17392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:43:07.785054 17392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:07.792783 17392 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:32853
I20260929 18:43:07.792838 17519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:32853 every 8 connection(s)
I20260929 18:43:07.793648 17392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:43:07.797189 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 17392
I20260929 18:43:07.797276 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:43:07.797501 7797 external_mini_cluster.cc:1428] Setting key 94d09815e59ba799082f0c876783cf27
I20260929 18:43:07.802621 17109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:07.795409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:45959 (local address 127.7.157.126:35461)
0929 18:43:07.795526 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:07.795530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:07.796639 (+ 1109us) server_negotiation.cc:407] Connection header received
0929 18:43:07.798080 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:07.798083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:07.798138 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:07.798212 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:07.799134 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:07.799664 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:07.800797 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:07.801027 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:07.801112 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:07.802160 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:07.802163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:07.802166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:43:07.802179 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:07.802200 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:43:07.802207 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:07.802263 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:07.802419 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 18:43:07.802481 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:43:07.803588 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:07.795848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:35461 (local address 127.7.157.67:45959)
0929 18:43:07.796492 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:07.796531 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:07.797866 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:07.798395 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:07.798403 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:07.798492 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:07.798993 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:07.799007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:07.799778 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:07.799782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:07.800678 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:07.800687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:07.801162 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:07.801166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:07.801345 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:07.801902 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:43:07.801903 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:07.801945 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:43:07.802008 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:43:07.802270 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:43:07.802279 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:43:07.802284 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:43:07.802417 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 18:43:07.802704 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20260929 18:43:07.804580 17520 heartbeater.cc:347] Connected to a master server at 127.7.157.126:35461
I20260929 18:43:07.804805 17520 heartbeater.cc:464] Registering TS with master...
I20260929 18:43:07.805299 17520 heartbeater.cc:511] Master 127.7.157.126:35461 requested a full tablet report, sending...
I20260929 18:43:07.806243 17070 ts_manager.cc:194] Registered new tserver with Master: 6443a98fd82d49168d6651833e034818 (127.7.157.67:32853)
I20260929 18:43:07.806784 17070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.157.67:45959
I20260929 18:43:07.811090 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:43:07.813687 17109 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:07.812847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.7.157.126:35461)
0929 18:43:07.812969 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:07.812974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:07.813144 (+ 170us) server_negotiation.cc:407] Connection header received
0929 18:43:07.813238 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:07.813240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:07.813280 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:07.813348 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:07.813568 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52362: BlockingRecv error: recv got EOF from 127.0.0.1:52362 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:43:07.813834 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 17130
I20260929 18:43:07.818463 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 17261
I20260929 18:43:07.823163 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 17392
I20260929 18:43:07.827415 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 17039
2026-09-29T18:43:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (830 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T18:43:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:43:07Z Disabled control of system clock
I20260929 18:43:07.854671 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:42727
--webserver_interface=127.7.157.126
--webserver_port=0
--builtin_ntp_servers=127.7.157.84:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:42727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:43:07.963191 17531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:43:07.963470 17531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:43:07.963507 17531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:43:07.967506 17531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:43:07.967561 17531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:43:07.967574 17531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:43:07.967590 17531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:43:07.967603 17531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:43:07.972370 17531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:42727
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:42727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:43:07.973378 17531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:43:07.974740 17531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:43:07.982969 17536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:07.982985 17537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:07.982982 17539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:07.983130 17531 server_base.cc:1034] running on GCE node
I20260929 18:43:07.983794 17531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:43:07.984879 17531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:43:07.986027 17531 hybrid_clock.cc:648] HybridClock initialized: now 1790707387986008 us; error 33 us; skew 500 ppm
I20260929 18:43:07.987808 17531 webserver.cc:460] Webserver started at http://127.7.157.126:42917/ using document root <none> and password file <none>
I20260929 18:43:07.988371 17531 fs_manager.cc:359] Metadata directory not provided
I20260929 18:43:07.988433 17531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:43:07.988616 17531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:43:07.990737 17531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "a3847b41ae534a70bd7440e8e6cc6fcd"
format_stamp: "Formatted at 2026-09-29 18:43:07 on dist-test-slave-smpn"
server_key: "0cbc9bf42aed3a4dda833bd88cf67565"
server_key_iv: "1a3185ba8472545cc28275374b8d7081"
server_key_version: "encryptionkey@0"
I20260929 18:43:07.991314 17531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "a3847b41ae534a70bd7440e8e6cc6fcd"
format_stamp: "Formatted at 2026-09-29 18:43:07 on dist-test-slave-smpn"
server_key: "0cbc9bf42aed3a4dda833bd88cf67565"
server_key_iv: "1a3185ba8472545cc28275374b8d7081"
server_key_version: "encryptionkey@0"
I20260929 18:43:07.994974 17531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:43:07.997357 17545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:07.998605 17531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:43:07.998723 17531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "a3847b41ae534a70bd7440e8e6cc6fcd"
format_stamp: "Formatted at 2026-09-29 18:43:07 on dist-test-slave-smpn"
server_key: "0cbc9bf42aed3a4dda833bd88cf67565"
server_key_iv: "1a3185ba8472545cc28275374b8d7081"
server_key_version: "encryptionkey@0"
I20260929 18:43:07.998828 17531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:43:08.003525 17531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:43:08.004143 17531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:43:08.004304 17531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:43:08.010838 17531 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:42727
I20260929 18:43:08.010859 17597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:42727 every 8 connection(s)
I20260929 18:43:08.011832 17531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:43:08.014986 17598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:43:08.020193 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 17531
I20260929 18:43:08.020300 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:43:08.020541 7797 external_mini_cluster.cc:1428] Setting key 2696b1de00c71067f0a911f2a6dc5f4f
I20260929 18:43:08.020625 17598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd: Bootstrap starting.
I20260929 18:43:08.023176 17598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd: Neither blocks nor log segments found. Creating new log.
I20260929 18:43:08.023875 17598 log.cc:824] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:43:08.025799 17598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd: No bootstrap required, opened a new log
I20260929 18:43:08.025961 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:08.021792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36966 (local address 127.7.157.126:42727)
0929 18:43:08.022456 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:08.022473 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:08.022509 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:43:08.023197 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:08.023216 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:08.023604 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:08.023924 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:08.024670 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:08.024675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:08.024687 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:43:08.024727 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:08.024785 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:43:08.024802 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:08.024892 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:08.025006 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 18:43:08.025244 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":126,"threads_started":1}
I20260929 18:43:08.029589 17598 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3847b41ae534a70bd7440e8e6cc6fcd" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 42727 } }
I20260929 18:43:08.029808 17598 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:43:08.029863 17598 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3847b41ae534a70bd7440e8e6cc6fcd, State: Initialized, Role: FOLLOWER
I20260929 18:43:08.030252 17598 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3847b41ae534a70bd7440e8e6cc6fcd" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 42727 } }
I20260929 18:43:08.030371 17598 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:43:08.030431 17598 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:43:08.030515 17598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:43:08.031418 17598 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3847b41ae534a70bd7440e8e6cc6fcd" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 42727 } }
I20260929 18:43:08.031723 17598 leader_election.cc:302] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3847b41ae534a70bd7440e8e6cc6fcd; no voters:
I20260929 18:43:08.032007 17598 leader_election.cc:288] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:43:08.032085 17603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:43:08.032294 17603 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [term 1 LEADER]: Becoming Leader. State: Replica: a3847b41ae534a70bd7440e8e6cc6fcd, State: Running, Role: LEADER
I20260929 18:43:08.032572 17603 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3847b41ae534a70bd7440e8e6cc6fcd" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 42727 } }
I20260929 18:43:08.033128 17598 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:43:08.034034 17604 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3847b41ae534a70bd7440e8e6cc6fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3847b41ae534a70bd7440e8e6cc6fcd" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 42727 } } }
I20260929 18:43:08.034158 17604 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [sys.catalog]: This master's current role is: LEADER
I20260929 18:43:08.034459 17612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:43:08.034416 17605 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3847b41ae534a70bd7440e8e6cc6fcd. Latest consensus state: current_term: 1 leader_uuid: "a3847b41ae534a70bd7440e8e6cc6fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3847b41ae534a70bd7440e8e6cc6fcd" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 42727 } } }
I20260929 18:43:08.034519 17605 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd [sys.catalog]: This master's current role is: LEADER
I20260929 18:43:08.037540 17612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:43:08.043256 17612 catalog_manager.cc:1347] Generated new cluster ID: f9a0d0e71e5b406bbf80b8a3e26a4d24
I20260929 18:43:08.043325 17612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:43:08.054515 17612 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:43:08.054809 17612 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:43:08.060571 17612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3847b41ae534a70bd7440e8e6cc6fcd: Generated new TSK 0
I20260929 18:43:08.061246 17612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:43:08.066030 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:42727
--builtin_ntp_servers=127.7.157.84:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:43:08.188802 17622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:43:08.189043 17622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:43:08.189081 17622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:43:08.193056 17622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:43:08.193110 17622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:43:08.193178 17622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:43:08.197885 17622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:42727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:43:08.198904 17622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:43:08.200297 17622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:43:08.209367 17622 server_base.cc:1034] running on GCE node
W20260929 18:43:08.209345 17627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:08.209323 17628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:08.209323 17630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:08.209945 17622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:43:08.210557 17622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:43:08.211714 17622 hybrid_clock.cc:648] HybridClock initialized: now 1790707388211691 us; error 31 us; skew 500 ppm
I20260929 18:43:08.213464 17622 webserver.cc:460] Webserver started at http://127.7.157.65:43423/ using document root <none> and password file <none>
I20260929 18:43:08.214020 17622 fs_manager.cc:359] Metadata directory not provided
I20260929 18:43:08.214085 17622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:43:08.214283 17622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:43:08.216337 17622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "7347fc9be40741d8ab3ab28814618fb1"
format_stamp: "Formatted at 2026-09-29 18:43:08 on dist-test-slave-smpn"
server_key: "80cd3299660c274b02f02c420cf1deb2"
server_key_iv: "d75f805a839475706f28225c3a45a08e"
server_key_version: "encryptionkey@0"
I20260929 18:43:08.216863 17622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "7347fc9be40741d8ab3ab28814618fb1"
format_stamp: "Formatted at 2026-09-29 18:43:08 on dist-test-slave-smpn"
server_key: "80cd3299660c274b02f02c420cf1deb2"
server_key_iv: "d75f805a839475706f28225c3a45a08e"
server_key_version: "encryptionkey@0"
I20260929 18:43:08.220444 17622 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:43:08.222862 17636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:08.224100 17622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:43:08.224202 17622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "7347fc9be40741d8ab3ab28814618fb1"
format_stamp: "Formatted at 2026-09-29 18:43:08 on dist-test-slave-smpn"
server_key: "80cd3299660c274b02f02c420cf1deb2"
server_key_iv: "d75f805a839475706f28225c3a45a08e"
server_key_version: "encryptionkey@0"
I20260929 18:43:08.224299 17622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:43:08.228868 17622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:43:08.229449 17622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:43:08.229600 17622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:43:08.230120 17622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:43:08.230983 17622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:43:08.231027 17622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:08.231106 17622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:43:08.231135 17622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:08.239213 17622 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:33169
I20260929 18:43:08.239243 17749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:33169 every 8 connection(s)
I20260929 18:43:08.240186 17622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:43:08.242180 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 17622
I20260929 18:43:08.242283 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:43:08.242561 7797 external_mini_cluster.cc:1428] Setting key aae718b34c260d6128da066826dbf498
I20260929 18:43:08.244426 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:42727
--builtin_ntp_servers=127.7.157.84:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 18:43:08.246503 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:08.242011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:40293 (local address 127.7.157.126:42727)
0929 18:43:08.242168 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:08.242173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:08.243457 (+ 1284us) server_negotiation.cc:407] Connection header received
0929 18:43:08.244753 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:08.244757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:08.244800 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:08.244885 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:08.246100 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:08.246104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:08.246107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:43:08.246121 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:08.246145 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:43:08.246154 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:08.246225 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:08.246362 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 18:43:08.246399 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:43:08.247545 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:08.242516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:42727 (local address 127.7.157.65:40293)
0929 18:43:08.243291 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:08.243354 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:08.244524 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:08.245046 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:08.245054 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:08.245148 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:08.245763 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:43:08.245765 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:08.245836 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:43:08.245932 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:43:08.246227 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:43:08.246234 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:43:08.246240 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:43:08.246412 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 18:43:08.246677 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260929 18:43:08.248746 17750 heartbeater.cc:347] Connected to a master server at 127.7.157.126:42727
I20260929 18:43:08.249006 17750 heartbeater.cc:464] Registering TS with master...
I20260929 18:43:08.249509 17750 heartbeater.cc:511] Master 127.7.157.126:42727 requested a full tablet report, sending...
I20260929 18:43:08.251205 17562 ts_manager.cc:194] Registered new tserver with Master: 7347fc9be40741d8ab3ab28814618fb1 (127.7.157.65:33169)
W20260929 18:43:08.358217 17753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:43:08.358618 17753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:43:08.358682 17753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:43:08.362782 17753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:43:08.362840 17753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:43:08.362896 17753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:43:08.367702 17753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:42727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:43:08.368770 17753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:43:08.370098 17753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:43:08.379387 17761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:08.379387 17758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:08.379387 17759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:08.379427 17753 server_base.cc:1034] running on GCE node
I20260929 18:43:08.380096 17753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:43:08.380713 17753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:43:08.381887 17753 hybrid_clock.cc:648] HybridClock initialized: now 1790707388381866 us; error 33 us; skew 500 ppm
I20260929 18:43:08.383749 17753 webserver.cc:460] Webserver started at http://127.7.157.66:43805/ using document root <none> and password file <none>
I20260929 18:43:08.384320 17753 fs_manager.cc:359] Metadata directory not provided
I20260929 18:43:08.384387 17753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:43:08.384572 17753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:43:08.386619 17753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "d2b33e03772e400eb526209bc85ad11c"
format_stamp: "Formatted at 2026-09-29 18:43:08 on dist-test-slave-smpn"
server_key: "5b7e019f38e0c84faf7c776629d90fdc"
server_key_iv: "1e4d9f95d9525b9b9777eafe01927e88"
server_key_version: "encryptionkey@0"
I20260929 18:43:08.387195 17753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "d2b33e03772e400eb526209bc85ad11c"
format_stamp: "Formatted at 2026-09-29 18:43:08 on dist-test-slave-smpn"
server_key: "5b7e019f38e0c84faf7c776629d90fdc"
server_key_iv: "1e4d9f95d9525b9b9777eafe01927e88"
server_key_version: "encryptionkey@0"
I20260929 18:43:08.390830 17753 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:43:08.393428 17767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:08.394658 17753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:43:08.394785 17753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "d2b33e03772e400eb526209bc85ad11c"
format_stamp: "Formatted at 2026-09-29 18:43:08 on dist-test-slave-smpn"
server_key: "5b7e019f38e0c84faf7c776629d90fdc"
server_key_iv: "1e4d9f95d9525b9b9777eafe01927e88"
server_key_version: "encryptionkey@0"
I20260929 18:43:08.394891 17753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:43:08.399426 17753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:43:08.400029 17753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:43:08.400187 17753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:43:08.400705 17753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:43:08.401583 17753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:43:08.401629 17753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:08.401677 17753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:43:08.401705 17753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:08.409981 17753 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:33275
I20260929 18:43:08.410001 17880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:33275 every 8 connection(s)
I20260929 18:43:08.410873 17753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:43:08.413702 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 17753
I20260929 18:43:08.413822 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:43:08.414094 7797 external_mini_cluster.cc:1428] Setting key 71542bb512cae26585565d4c03f325f6
I20260929 18:43:08.416105 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:42727
--builtin_ntp_servers=127.7.157.84:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 18:43:08.417009 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:08.412641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:42239 (local address 127.7.157.126:42727)
0929 18:43:08.412776 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:08.412781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:08.413922 (+ 1141us) server_negotiation.cc:407] Connection header received
0929 18:43:08.415330 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:08.415333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:08.415369 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:08.415450 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:08.416628 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:08.416631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:08.416633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:43:08.416643 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:08.416662 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:43:08.416670 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:08.416738 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:08.416873 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 18:43:08.416907 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:43:08.417982 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:08.413088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:42727 (local address 127.7.157.66:42239)
0929 18:43:08.413762 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:08.413815 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:08.415122 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:08.415642 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:08.415656 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:08.415770 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:08.416337 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:43:08.416339 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:08.416392 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:43:08.416481 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:43:08.416740 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:43:08.416746 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:43:08.416750 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:43:08.416915 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 18:43:08.417148 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260929 18:43:08.419045 17881 heartbeater.cc:347] Connected to a master server at 127.7.157.126:42727
I20260929 18:43:08.419296 17881 heartbeater.cc:464] Registering TS with master...
I20260929 18:43:08.419668 17881 heartbeater.cc:511] Master 127.7.157.126:42727 requested a full tablet report, sending...
I20260929 18:43:08.420629 17562 ts_manager.cc:194] Registered new tserver with Master: d2b33e03772e400eb526209bc85ad11c (127.7.157.66:33275)
W20260929 18:43:08.524869 17884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:43:08.525130 17884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:43:08.525171 17884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:43:08.529278 17884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:43:08.529340 17884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:43:08.529400 17884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:43:08.534206 17884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:42727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:43:08.535338 17884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:43:08.536641 17884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:43:08.545904 17884 server_base.cc:1034] running on GCE node
W20260929 18:43:08.545881 17890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:08.545993 17889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:08.545884 17892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:08.546509 17884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:43:08.547206 17884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:43:08.548364 17884 hybrid_clock.cc:648] HybridClock initialized: now 1790707388548334 us; error 34 us; skew 500 ppm
I20260929 18:43:08.550100 17884 webserver.cc:460] Webserver started at http://127.7.157.67:45323/ using document root <none> and password file <none>
I20260929 18:43:08.550648 17884 fs_manager.cc:359] Metadata directory not provided
I20260929 18:43:08.550712 17884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:43:08.550895 17884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:43:08.552981 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "e2df11d662b448249e44c6cea7cc81fc"
format_stamp: "Formatted at 2026-09-29 18:43:08 on dist-test-slave-smpn"
server_key: "7fe29cc854cc061d67f86bc2b56d753a"
server_key_iv: "b018309c9bcea9d840847811bde84a59"
server_key_version: "encryptionkey@0"
I20260929 18:43:08.553515 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "e2df11d662b448249e44c6cea7cc81fc"
format_stamp: "Formatted at 2026-09-29 18:43:08 on dist-test-slave-smpn"
server_key: "7fe29cc854cc061d67f86bc2b56d753a"
server_key_iv: "b018309c9bcea9d840847811bde84a59"
server_key_version: "encryptionkey@0"
I20260929 18:43:08.557102 17884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 18:43:08.559554 17898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:08.560815 17884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:43:08.560916 17884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "e2df11d662b448249e44c6cea7cc81fc"
format_stamp: "Formatted at 2026-09-29 18:43:08 on dist-test-slave-smpn"
server_key: "7fe29cc854cc061d67f86bc2b56d753a"
server_key_iv: "b018309c9bcea9d840847811bde84a59"
server_key_version: "encryptionkey@0"
I20260929 18:43:08.561010 17884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:43:08.565827 17884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:43:08.566483 17884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:43:08.566627 17884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:43:08.567140 17884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:43:08.567991 17884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:43:08.568044 17884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:08.568092 17884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:43:08.568115 17884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:08.576435 17884 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:37455
I20260929 18:43:08.576459 18011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:37455 every 8 connection(s)
I20260929 18:43:08.577309 17884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:43:08.582355 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 17884
I20260929 18:43:08.582453 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:43:08.582706 7797 external_mini_cluster.cc:1428] Setting key 55c8b6e27ee62c374dd241e89f475f10
I20260929 18:43:08.583250 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:08.579028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:53567 (local address 127.7.157.126:42727)
0929 18:43:08.579173 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:08.579177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:08.580306 (+ 1129us) server_negotiation.cc:407] Connection header received
0929 18:43:08.581578 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:08.581582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:08.581627 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:08.581699 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:08.582802 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:08.582806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:08.582809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:43:08.582822 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:08.582845 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:43:08.582856 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:08.582936 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:08.583056 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 18:43:08.583121 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 18:43:08.584275 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:08.579467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:42727 (local address 127.7.157.67:53567)
0929 18:43:08.580169 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:08.580209 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:08.581373 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:08.581881 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:08.581888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:08.581978 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:08.582548 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:43:08.582550 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:08.582592 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:43:08.582660 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:43:08.582928 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:43:08.582936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:43:08.582942 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:43:08.583131 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 18:43:08.583372 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":108,"threads_started":1}
I20260929 18:43:08.585245 18012 heartbeater.cc:347] Connected to a master server at 127.7.157.126:42727
I20260929 18:43:08.585440 18012 heartbeater.cc:464] Registering TS with master...
I20260929 18:43:08.585836 18012 heartbeater.cc:511] Master 127.7.157.126:42727 requested a full tablet report, sending...
I20260929 18:43:08.586720 17562 ts_manager.cc:194] Registered new tserver with Master: e2df11d662b448249e44c6cea7cc81fc (127.7.157.67:37455)
I20260929 18:43:08.596554 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:43:08.599778 17601 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:08.598535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36970 (local address 127.7.157.126:42727)
0929 18:43:08.598736 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:08.598741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:08.598979 (+ 238us) server_negotiation.cc:407] Connection header received
0929 18:43:08.599159 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:08.599162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:08.599215 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:08.599299 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:08.599581 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36970: BlockingRecv error: recv got EOF from 127.0.0.1:36970 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 18:43:08.599823 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 17622
I20260929 18:43:08.605093 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 17753
I20260929 18:43:08.609359 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 17884
I20260929 18:43:08.613775 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 17531
2026-09-29T18:43:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (785 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:43:08 dist-test-slave-smpn krb5kdc[18022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:43:08 dist-test-slave-smpn krb5kdc[18022](info): set up 2 sockets
Sep 29 18:43:08 dist-test-slave-smpn krb5kdc[18022](info): commencing operation
krb5kdc: starting...
W20260929 18:43:09.567679 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:43:09 dist-test-slave-smpn krb5kdc[18022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:43:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:43:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:43:09.686655 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:44367
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:44367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:43:09.796134 18035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:43:09.796396 18035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:43:09.796434 18035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:43:09.800397 18035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:43:09.800462 18035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:43:09.800480 18035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:43:09.800495 18035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:43:09.800509 18035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:43:09.805928 18035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:44367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:44367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:43:09.806994 18035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:43:09.808072 18035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:43:09.816383 18040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:09.816440 18041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:09.816504 18035 server_base.cc:1034] running on GCE node
W20260929 18:43:09.816408 18043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:09.817260 18035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:43:09.818356 18035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:43:09.819540 18035 hybrid_clock.cc:648] HybridClock initialized: now 1790707389819521 us; error 29 us; skew 500 ppm
Sep 29 18:43:09 dist-test-slave-smpn krb5kdc[18022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707389, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:43:09.822194 18035 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:43:09.823061 18035 webserver.cc:460] Webserver started at http://127.7.157.126:39825/ using document root <none> and password file <none>
I20260929 18:43:09.823679 18035 fs_manager.cc:359] Metadata directory not provided
I20260929 18:43:09.823750 18035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:43:09.823962 18035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:43:09.825948 18035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "aed2d16a37104e7a9d9e492412793c96"
format_stamp: "Formatted at 2026-09-29 18:43:09 on dist-test-slave-smpn"
server_key: "ca336e0458a0e939a1968b8e84de2671"
server_key_iv: "8fa6d41770bdae25e95566ac57014b1d"
server_key_version: "encryptionkey@0"
I20260929 18:43:09.826463 18035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "aed2d16a37104e7a9d9e492412793c96"
format_stamp: "Formatted at 2026-09-29 18:43:09 on dist-test-slave-smpn"
server_key: "ca336e0458a0e939a1968b8e84de2671"
server_key_iv: "8fa6d41770bdae25e95566ac57014b1d"
server_key_version: "encryptionkey@0"
I20260929 18:43:09.830055 18035 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:43:09.832402 18050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:09.833626 18035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:43:09.833739 18035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "aed2d16a37104e7a9d9e492412793c96"
format_stamp: "Formatted at 2026-09-29 18:43:09 on dist-test-slave-smpn"
server_key: "ca336e0458a0e939a1968b8e84de2671"
server_key_iv: "8fa6d41770bdae25e95566ac57014b1d"
server_key_version: "encryptionkey@0"
I20260929 18:43:09.833834 18035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:43:09.851749 18035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:43:09.854758 18035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:43:09.854945 18035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:43:09.861740 18035 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:44367
I20260929 18:43:09.862001 18102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:44367 every 8 connection(s)
I20260929 18:43:09.862774 18035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:43:09.865832 18103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:43:09.871537 18103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96: Bootstrap starting.
I20260929 18:43:09.872756 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 18035
I20260929 18:43:09.872856 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:43:09.873095 7797 external_mini_cluster.cc:1428] Setting key e019442e728ac3138bbca1a4aef40c5b
I20260929 18:43:09.874164 18103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96: Neither blocks nor log segments found. Creating new log.
I20260929 18:43:09.874993 18103 log.cc:824] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96: Log is configured to *not* fsync() on all Append() calls
I20260929 18:43:09.876852 18103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96: No bootstrap required, opened a new log
I20260929 18:43:09.880342 18103 raft_consensus.cc:348] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed2d16a37104e7a9d9e492412793c96" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44367 } }
I20260929 18:43:09.880574 18103 raft_consensus.cc:374] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:43:09.880631 18103 raft_consensus.cc:729] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed2d16a37104e7a9d9e492412793c96, State: Initialized, Role: FOLLOWER
I20260929 18:43:09.881063 18103 consensus_queue.cc:260] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed2d16a37104e7a9d9e492412793c96" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44367 } }
I20260929 18:43:09.881225 18103 raft_consensus.cc:388] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:43:09.881301 18103 raft_consensus.cc:482] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:43:09.881403 18103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 18:43:09 dist-test-slave-smpn krb5kdc[18022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:43:09.882308 18103 raft_consensus.cc:504] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed2d16a37104e7a9d9e492412793c96" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44367 } }
I20260929 18:43:09.882673 18103 leader_election.cc:302] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aed2d16a37104e7a9d9e492412793c96; no voters:
I20260929 18:43:09.882987 18103 leader_election.cc:288] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:43:09.883060 18108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:43:09.884111 18108 raft_consensus.cc:686] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [term 1 LEADER]: Becoming Leader. State: Replica: aed2d16a37104e7a9d9e492412793c96, State: Running, Role: LEADER
I20260929 18:43:09.884209 18103 sys_catalog.cc:564] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:43:09.884466 18108 consensus_queue.cc:237] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed2d16a37104e7a9d9e492412793c96" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44367 } }
I20260929 18:43:09.886086 18109 sys_catalog.cc:455] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aed2d16a37104e7a9d9e492412793c96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed2d16a37104e7a9d9e492412793c96" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44367 } } }
I20260929 18:43:09.886209 18109 sys_catalog.cc:458] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [sys.catalog]: This master's current role is: LEADER
I20260929 18:43:09.886540 18110 sys_catalog.cc:455] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aed2d16a37104e7a9d9e492412793c96. Latest consensus state: current_term: 1 leader_uuid: "aed2d16a37104e7a9d9e492412793c96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed2d16a37104e7a9d9e492412793c96" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44367 } } }
I20260929 18:43:09.886725 18110 sys_catalog.cc:458] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96 [sys.catalog]: This master's current role is: LEADER
I20260929 18:43:09.886835 18123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:43:09.887878 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:09.874232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46726 (local address 127.7.157.126:44367)
0929 18:43:09.874737 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:09.874754 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:09.874790 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:43:09.875517 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:09.875537 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:09.875898 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:09.876216 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:09.876998 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:09.877985 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:09.878824 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:09.879351 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:09.879500 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:09.882137 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:09.882142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:09.882156 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:43:09.882184 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:09.883875 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:09.884467 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:09.884473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:09.884479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:09.884635 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:09.884962 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:09.884966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:09.884968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:09.885538 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:09.885739 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:09.886523 (+ 784us) server_negotiation.cc:299] Negotiation successful
0929 18:43:09.886845 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":154,"threads_started":1}
I20260929 18:43:09.889798 18123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:43:09.895303 18123 catalog_manager.cc:1347] Generated new cluster ID: a6f815637b6543baab194bb87c8e107e
I20260929 18:43:09.895357 18123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:43:09.909611 18123 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:43:09.910589 18123 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:43:09.914222 18123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96: Generated new TSK 0
I20260929 18:43:09.915139 18123 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:43:09.949795 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:44367
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:43:10.060071 18129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:43:10.060356 18129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:43:10.060415 18129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:43:10.064554 18129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:43:10.064626 18129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:43:10.064709 18129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:43:10.070039 18129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:44367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:43:10.071169 18129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:43:10.072198 18129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:43:10.081571 18134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:10.081598 18135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:10.081701 18129 server_base.cc:1034] running on GCE node
W20260929 18:43:10.081573 18137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:10.082233 18129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:43:10.082873 18129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:43:10.084061 18129 hybrid_clock.cc:648] HybridClock initialized: now 1790707390084035 us; error 35 us; skew 500 ppm
Sep 29 18:43:10 dist-test-slave-smpn krb5kdc[18022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707390, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:43:10.086843 18129 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:43:10.087742 18129 webserver.cc:460] Webserver started at http://127.7.157.65:43165/ using document root <none> and password file <none>
I20260929 18:43:10.088259 18129 fs_manager.cc:359] Metadata directory not provided
I20260929 18:43:10.088315 18129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:43:10.088485 18129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:43:10.090526 18129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "555cc6035ad646d3a1b7dc0cda437285"
format_stamp: "Formatted at 2026-09-29 18:43:10 on dist-test-slave-smpn"
server_key: "26a0425123438aed204a8bad77ae4803"
server_key_iv: "1c61a2b45b0f72a18e60d7d593ecbb9a"
server_key_version: "encryptionkey@0"
I20260929 18:43:10.091039 18129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "555cc6035ad646d3a1b7dc0cda437285"
format_stamp: "Formatted at 2026-09-29 18:43:10 on dist-test-slave-smpn"
server_key: "26a0425123438aed204a8bad77ae4803"
server_key_iv: "1c61a2b45b0f72a18e60d7d593ecbb9a"
server_key_version: "encryptionkey@0"
I20260929 18:43:10.094741 18129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:43:10.097133 18144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:10.098356 18129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:43:10.098453 18129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "555cc6035ad646d3a1b7dc0cda437285"
format_stamp: "Formatted at 2026-09-29 18:43:10 on dist-test-slave-smpn"
server_key: "26a0425123438aed204a8bad77ae4803"
server_key_iv: "1c61a2b45b0f72a18e60d7d593ecbb9a"
server_key_version: "encryptionkey@0"
I20260929 18:43:10.098553 18129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:43:10.108608 18129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:43:10.112047 18129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:43:10.112219 18129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:43:10.112787 18129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:43:10.113581 18129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:43:10.113631 18129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:10.113682 18129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:43:10.113765 18129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:10.122285 18129 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:39661
I20260929 18:43:10.122306 18257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:39661 every 8 connection(s)
I20260929 18:43:10.123281 18129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:43:10.126336 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 18129
I20260929 18:43:10.126454 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:43:10.126716 7797 external_mini_cluster.cc:1428] Setting key 0c8a687b0969a0c70a60a1875d846229
Sep 29 18:43:10 dist-test-slave-smpn krb5kdc[18022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707390, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
I20260929 18:43:10.137050 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.125123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:40747 (local address 127.7.157.126:44367)
0929 18:43:10.125299 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:10.125304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:10.126209 (+ 905us) server_negotiation.cc:407] Connection header received
0929 18:43:10.127886 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:10.127889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:10.127942 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:10.128014 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:10.129245 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.129772 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.130924 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.131201 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.131289 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.133625 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:10.133629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:10.133631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:43:10.133655 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:10.135013 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:10.135520 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:10.135523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:10.135525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:10.135601 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:10.135923 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:10.135926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:10.135928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:10.136137 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:10.136238 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:10.136713 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 18:43:10.136877 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
I20260929 18:43:10.137889 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.125563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:44367 (local address 127.7.157.65:40747)
0929 18:43:10.126041 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:10.126084 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:10.127675 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:10.128202 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:10.128210 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:10.128585 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:10.129091 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:10.129105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.129891 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.129894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:10.130807 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:10.130816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.131345 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.131349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:10.131521 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.132088 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:10.132105 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:10.133462 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:43:10.135151 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:43:10.135157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:43:10.135175 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:43:10.135427 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:43:10.135692 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:43:10.135695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:43:10.135697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:43:10.135837 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:43:10.136243 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:43:10.136251 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:43:10.136493 (+ 242us) client_negotiation.cc:769] Sending connection context
0929 18:43:10.136667 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 18:43:10.137019 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":132,"threads_started":1}
I20260929 18:43:10.139302 18258 heartbeater.cc:347] Connected to a master server at 127.7.157.126:44367
I20260929 18:43:10.139564 18258 heartbeater.cc:464] Registering TS with master...
I20260929 18:43:10.140123 18258 heartbeater.cc:511] Master 127.7.157.126:44367 requested a full tablet report, sending...
I20260929 18:43:10.141616 18067 ts_manager.cc:194] Registered new tserver with Master: 555cc6035ad646d3a1b7dc0cda437285 (127.7.157.65:39661)
I20260929 18:43:10.142714 18067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:40747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:43:10.153776 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:44367
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:43:10.262841 18263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:43:10.263170 18263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:43:10.263221 18263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:43:10.267309 18263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:43:10.267375 18263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:43:10.267438 18263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:43:10.272769 18263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:44367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:43:10.273803 18263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:43:10.274812 18263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:43:10.284555 18268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:10.284538 18269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:10.284547 18271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:10.284901 18263 server_base.cc:1034] running on GCE node
I20260929 18:43:10.285336 18263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:43:10.285962 18263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:43:10.287164 18263 hybrid_clock.cc:648] HybridClock initialized: now 1790707390287119 us; error 48 us; skew 500 ppm
Sep 29 18:43:10 dist-test-slave-smpn krb5kdc[18022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707390, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:43:10.289880 18263 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:43:10.290793 18263 webserver.cc:460] Webserver started at http://127.7.157.66:44409/ using document root <none> and password file <none>
I20260929 18:43:10.291375 18263 fs_manager.cc:359] Metadata directory not provided
I20260929 18:43:10.291436 18263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:43:10.291620 18263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:43:10.293720 18263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "d7d1e2d1cbf944dfb8ec3c087dcd7f1f"
format_stamp: "Formatted at 2026-09-29 18:43:10 on dist-test-slave-smpn"
server_key: "e8342162a476a2fa5608f6d473f6e970"
server_key_iv: "eb90eb0bb180b8ad14f6ec422d450707"
server_key_version: "encryptionkey@0"
I20260929 18:43:10.294255 18263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "d7d1e2d1cbf944dfb8ec3c087dcd7f1f"
format_stamp: "Formatted at 2026-09-29 18:43:10 on dist-test-slave-smpn"
server_key: "e8342162a476a2fa5608f6d473f6e970"
server_key_iv: "eb90eb0bb180b8ad14f6ec422d450707"
server_key_version: "encryptionkey@0"
I20260929 18:43:10.298383 18263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:43:10.301412 18278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:10.302927 18263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:43:10.303038 18263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "d7d1e2d1cbf944dfb8ec3c087dcd7f1f"
format_stamp: "Formatted at 2026-09-29 18:43:10 on dist-test-slave-smpn"
server_key: "e8342162a476a2fa5608f6d473f6e970"
server_key_iv: "eb90eb0bb180b8ad14f6ec422d450707"
server_key_version: "encryptionkey@0"
I20260929 18:43:10.303170 18263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:43:10.323405 18263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:43:10.326822 18263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:43:10.326992 18263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:43:10.327622 18263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:43:10.328532 18263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:43:10.328578 18263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:10.328619 18263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:43:10.328646 18263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:10.336882 18263 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:42445
I20260929 18:43:10.336917 18391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:42445 every 8 connection(s)
I20260929 18:43:10.337867 18263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:43:10.339980 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 18263
I20260929 18:43:10.340088 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:43:10.340389 7797 external_mini_cluster.cc:1428] Setting key c21e0b488e5c88d07c22dcfe59dcc35a
Sep 29 18:43:10 dist-test-slave-smpn krb5kdc[18022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707390, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
I20260929 18:43:10.351958 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.339690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:37019 (local address 127.7.157.126:44367)
0929 18:43:10.339842 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:10.339846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:10.340924 (+ 1078us) server_negotiation.cc:407] Connection header received
0929 18:43:10.342378 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:10.342393 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:10.342444 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:10.342539 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:10.343960 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.344489 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.345668 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.345901 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.345991 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.348579 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:10.348582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:10.348584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:43:10.348608 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:10.349895 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:10.350413 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:10.350417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:10.350419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:10.350493 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:10.350803 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:10.350807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:10.350809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:10.351009 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:10.351141 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:10.351686 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 18:43:10.351801 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:43:10.352831 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.340198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:44367 (local address 127.7.157.66:37019)
0929 18:43:10.340736 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:10.340798 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:10.342160 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:10.342697 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:10.342705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:10.343098 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:10.343793 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:10.343813 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.344602 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.344605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:10.345538 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:10.345548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.346045 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.346049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:10.346218 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.346750 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:10.346766 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:10.348414 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:43:10.350028 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:43:10.350033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:43:10.350050 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:43:10.350319 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:43:10.350572 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:43:10.350575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:43:10.350577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:43:10.350719 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:43:10.351129 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:43:10.351135 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:43:10.351387 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 18:43:10.351614 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 18:43:10.351967 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":157,"threads_started":1}
I20260929 18:43:10.353986 18392 heartbeater.cc:347] Connected to a master server at 127.7.157.126:44367
I20260929 18:43:10.354228 18392 heartbeater.cc:464] Registering TS with master...
I20260929 18:43:10.354694 18392 heartbeater.cc:511] Master 127.7.157.126:44367 requested a full tablet report, sending...
I20260929 18:43:10.355737 18067 ts_manager.cc:194] Registered new tserver with Master: d7d1e2d1cbf944dfb8ec3c087dcd7f1f (127.7.157.66:42445)
I20260929 18:43:10.356514 18067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:37019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:43:10.366743 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:44367
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:43:10.474987 18397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:43:10.475276 18397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:43:10.475318 18397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:43:10.479249 18397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:43:10.479302 18397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:43:10.479358 18397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:43:10.484637 18397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:44367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:43:10.485778 18397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:43:10.486802 18397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:43:10.496234 18405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:10.496272 18402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:10.496264 18403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:10.496449 18397 server_base.cc:1034] running on GCE node
I20260929 18:43:10.496846 18397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:43:10.497458 18397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:43:10.498622 18397 hybrid_clock.cc:648] HybridClock initialized: now 1790707390498601 us; error 33 us; skew 500 ppm
Sep 29 18:43:10 dist-test-slave-smpn krb5kdc[18022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707390, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:43:10.501263 18397 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:43:10.502158 18397 webserver.cc:460] Webserver started at http://127.7.157.67:41573/ using document root <none> and password file <none>
I20260929 18:43:10.502708 18397 fs_manager.cc:359] Metadata directory not provided
I20260929 18:43:10.502772 18397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:43:10.502998 18397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:43:10.505095 18397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "37dcbd45cb734188bf4671270d7e6e24"
format_stamp: "Formatted at 2026-09-29 18:43:10 on dist-test-slave-smpn"
server_key: "69452d38a810bc1464780081d3effd4d"
server_key_iv: "ff66c5a32ecc36d2f14bb9568768f268"
server_key_version: "encryptionkey@0"
I20260929 18:43:10.505616 18397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "37dcbd45cb734188bf4671270d7e6e24"
format_stamp: "Formatted at 2026-09-29 18:43:10 on dist-test-slave-smpn"
server_key: "69452d38a810bc1464780081d3effd4d"
server_key_iv: "ff66c5a32ecc36d2f14bb9568768f268"
server_key_version: "encryptionkey@0"
I20260929 18:43:10.509378 18397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:43:10.511744 18412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:10.513001 18397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:43:10.513098 18397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "37dcbd45cb734188bf4671270d7e6e24"
format_stamp: "Formatted at 2026-09-29 18:43:10 on dist-test-slave-smpn"
server_key: "69452d38a810bc1464780081d3effd4d"
server_key_iv: "ff66c5a32ecc36d2f14bb9568768f268"
server_key_version: "encryptionkey@0"
I20260929 18:43:10.513191 18397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:43:10.534749 18397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:43:10.538254 18397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:43:10.538455 18397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:43:10.539088 18397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:43:10.539950 18397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:43:10.539999 18397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:10.540055 18397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:43:10.540083 18397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:10.548400 18397 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:41189
I20260929 18:43:10.548434 18525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:41189 every 8 connection(s)
I20260929 18:43:10.549329 18397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:43:10.552896 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 18397
I20260929 18:43:10.553006 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:43:10.553257 7797 external_mini_cluster.cc:1428] Setting key 436f0712823a963e4e522aabf9c5d767
Sep 29 18:43:10 dist-test-slave-smpn krb5kdc[18022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707390, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:43:10.562678 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.551137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:51601 (local address 127.7.157.126:44367)
0929 18:43:10.551278 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:10.551283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:10.552188 (+ 905us) server_negotiation.cc:407] Connection header received
0929 18:43:10.553556 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:10.553558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:10.553606 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:10.553681 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:10.554984 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.555593 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.556704 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.556924 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.557011 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.559412 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:10.559416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:10.559418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:43:10.559444 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:10.560718 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:10.561221 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:10.561224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:10.561225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:10.561302 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:10.561610 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:10.561613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:10.561615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:10.561789 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:10.561873 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:10.562372 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 18:43:10.562505 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:43:10.563594 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.551586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:44367 (local address 127.7.157.67:51601)
0929 18:43:10.552029 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:10.552074 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:10.553334 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:10.553875 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:10.553883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:10.554300 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:10.554844 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:10.554859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.555712 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.555715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:10.556589 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:10.556598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.557055 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.557059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:10.557220 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.557751 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:10.557767 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:10.559246 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:43:10.560865 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:43:10.560870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:43:10.560884 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:43:10.561127 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:43:10.561387 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:43:10.561391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:43:10.561392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:43:10.561525 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:43:10.561882 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:43:10.561888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:43:10.562138 (+ 250us) client_negotiation.cc:769] Sending connection context
0929 18:43:10.562310 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 18:43:10.562666 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":116,"threads_started":1}
I20260929 18:43:10.564685 18526 heartbeater.cc:347] Connected to a master server at 127.7.157.126:44367
I20260929 18:43:10.564920 18526 heartbeater.cc:464] Registering TS with master...
I20260929 18:43:10.565403 18526 heartbeater.cc:511] Master 127.7.157.126:44367 requested a full tablet report, sending...
I20260929 18:43:10.566471 18067 ts_manager.cc:194] Registered new tserver with Master: 37dcbd45cb734188bf4671270d7e6e24 (127.7.157.67:41189)
I20260929 18:43:10.566933 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:43:10.567168 18067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:51601
I20260929 18:43:10.575632 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.568820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46738 (local address 127.7.157.126:44367)
0929 18:43:10.568925 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:10.568930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:10.569103 (+ 173us) server_negotiation.cc:407] Connection header received
0929 18:43:10.569208 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:10.569211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:10.569252 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:10.569314 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:10.569949 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.570447 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.571302 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.571515 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.571596 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.572827 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:10.572831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:10.572834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:43:10.572858 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:10.574095 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:10.574485 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:10.574489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:10.574491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:10.574564 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:10.574798 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:10.574801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:10.574803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:10.575016 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:10.575158 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:10.575371 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 18:43:10.575482 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:43:10.578413 18067 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:43:10.581398 18067 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:43:10.594950 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.589378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:41189 (local address 127.0.0.1:33936)
0929 18:43:10.589733 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:10.589747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:10.589852 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:10.590946 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:10.590949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:10.590965 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:43:10.591058 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:10.591063 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.592565 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.592568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:10.593599 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:10.593608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.594392 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.594395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:10.594499 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.594535 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 18:43:10.594651 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 18:43:10.594777 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":37,"thread_start_us":90,"threads_started":1}
I20260929 18:43:10.595116 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.589344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59390 (local address 127.7.157.65:39661)
0929 18:43:10.590126 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:10.590138 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:10.590159 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:43:10.590234 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:10.590239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:10.590449 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:10.590954 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:43:10.591123 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.592308 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.593747 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.594560 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.594670 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.594806 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:10.594872 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 18:43:10.594952 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":133,"threads_started":1}
I20260929 18:43:10.595281 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.589975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33936 (local address 127.7.157.67:41189)
0929 18:43:10.590483 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:10.590493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:10.590515 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:43:10.590583 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:10.590588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:10.590769 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:10.591221 (+ 452us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:43:10.591373 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.592451 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.593729 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.594278 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.594383 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.594539 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:10.595018 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 18:43:10.595148 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":88,"threads_started":1}
I20260929 18:43:10.595467 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.589229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:39661 (local address 127.0.0.1:59390)
0929 18:43:10.589670 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:10.589716 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:10.589852 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:10.590615 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:10.590621 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:10.590643 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:43:10.590792 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:10.590799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.592440 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.592443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:10.593631 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:10.593638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.594677 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.594681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:10.594764 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.594784 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:43:10.594840 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:43:10.594898 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":33,"thread_start_us":79,"threads_started":1}
I20260929 18:43:10.596719 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.590342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34672 (local address 127.7.157.66:42445)
0929 18:43:10.590729 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:10.590743 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:10.590765 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:43:10.590836 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:10.590841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:10.591009 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:10.591168 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:43:10.591358 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.592055 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.593709 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.594560 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.594670 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.594806 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:10.596481 (+ 1675us) server_negotiation.cc:299] Negotiation successful
0929 18:43:10.596584 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":108,"threads_started":1}
I20260929 18:43:10.596909 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.589900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:42445 (local address 127.0.0.1:34672)
0929 18:43:10.590120 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:10.590133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:10.590211 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:10.591144 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:10.591148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:10.591165 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:43:10.591270 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:10.591277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.592169 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.592173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:10.593567 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:10.593578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.596074 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.596077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:10.596152 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.596335 (+ 183us) client_negotiation.cc:769] Sending connection context
0929 18:43:10.596415 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:43:10.596490 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":30,"thread_start_us":96,"threads_started":1}
I20260929 18:43:10.598268 18192 tablet_service.cc:1467] Processing CreateTablet for tablet 568c16b0936c496eab17a8bc8570f71b (DEFAULT_TABLE table=test-table [id=a98a7ea88b484f5e805ee968febfe860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:43:10.598275 18460 tablet_service.cc:1467] Processing CreateTablet for tablet 568c16b0936c496eab17a8bc8570f71b (DEFAULT_TABLE table=test-table [id=a98a7ea88b484f5e805ee968febfe860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:43:10.599284 18192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568c16b0936c496eab17a8bc8570f71b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:43:10.599306 18460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568c16b0936c496eab17a8bc8570f71b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:43:10.599648 18326 tablet_service.cc:1467] Processing CreateTablet for tablet 568c16b0936c496eab17a8bc8570f71b (DEFAULT_TABLE table=test-table [id=a98a7ea88b484f5e805ee968febfe860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:43:10.600582 18326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568c16b0936c496eab17a8bc8570f71b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:43:10.605182 18543 tablet_bootstrap.cc:492] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285: Bootstrap starting.
I20260929 18:43:10.607107 18545 tablet_bootstrap.cc:492] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f: Bootstrap starting.
I20260929 18:43:10.607434 18544 tablet_bootstrap.cc:492] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24: Bootstrap starting.
I20260929 18:43:10.608282 18543 tablet_bootstrap.cc:654] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285: Neither blocks nor log segments found. Creating new log.
I20260929 18:43:10.609465 18545 tablet_bootstrap.cc:654] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f: Neither blocks nor log segments found. Creating new log.
I20260929 18:43:10.609568 18543 log.cc:824] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285: Log is configured to *not* fsync() on all Append() calls
I20260929 18:43:10.609895 18544 tablet_bootstrap.cc:654] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24: Neither blocks nor log segments found. Creating new log.
I20260929 18:43:10.610186 18545 log.cc:824] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:43:10.610597 18544 log.cc:824] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24: Log is configured to *not* fsync() on all Append() calls
I20260929 18:43:10.611711 18543 tablet_bootstrap.cc:492] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285: No bootstrap required, opened a new log
I20260929 18:43:10.611904 18543 ts_tablet_manager.cc:1397] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:43:10.612140 18545 tablet_bootstrap.cc:492] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f: No bootstrap required, opened a new log
I20260929 18:43:10.612414 18545 ts_tablet_manager.cc:1397] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:43:10.612437 18544 tablet_bootstrap.cc:492] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24: No bootstrap required, opened a new log
I20260929 18:43:10.612592 18544 ts_tablet_manager.cc:1397] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:43:10.615294 18545 raft_consensus.cc:348] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555cc6035ad646d3a1b7dc0cda437285" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39661 } } peers { permanent_uuid: "37dcbd45cb734188bf4671270d7e6e24" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 41189 } } peers { permanent_uuid: "d7d1e2d1cbf944dfb8ec3c087dcd7f1f" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42445 } }
I20260929 18:43:10.615512 18545 raft_consensus.cc:374] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:43:10.615563 18545 raft_consensus.cc:729] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d1e2d1cbf944dfb8ec3c087dcd7f1f, State: Initialized, Role: FOLLOWER
I20260929 18:43:10.615731 18543 raft_consensus.cc:348] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555cc6035ad646d3a1b7dc0cda437285" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39661 } } peers { permanent_uuid: "37dcbd45cb734188bf4671270d7e6e24" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 41189 } } peers { permanent_uuid: "d7d1e2d1cbf944dfb8ec3c087dcd7f1f" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42445 } }
I20260929 18:43:10.615979 18543 raft_consensus.cc:374] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:43:10.616034 18543 raft_consensus.cc:729] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555cc6035ad646d3a1b7dc0cda437285, State: Initialized, Role: FOLLOWER
I20260929 18:43:10.616000 18545 consensus_queue.cc:260] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555cc6035ad646d3a1b7dc0cda437285" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39661 } } peers { permanent_uuid: "37dcbd45cb734188bf4671270d7e6e24" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 41189 } } peers { permanent_uuid: "d7d1e2d1cbf944dfb8ec3c087dcd7f1f" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42445 } }
I20260929 18:43:10.616178 18544 raft_consensus.cc:348] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555cc6035ad646d3a1b7dc0cda437285" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39661 } } peers { permanent_uuid: "37dcbd45cb734188bf4671270d7e6e24" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 41189 } } peers { permanent_uuid: "d7d1e2d1cbf944dfb8ec3c087dcd7f1f" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42445 } }
I20260929 18:43:10.616411 18544 raft_consensus.cc:374] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:43:10.616463 18544 raft_consensus.cc:729] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dcbd45cb734188bf4671270d7e6e24, State: Initialized, Role: FOLLOWER
I20260929 18:43:10.616438 18543 consensus_queue.cc:260] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555cc6035ad646d3a1b7dc0cda437285" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39661 } } peers { permanent_uuid: "37dcbd45cb734188bf4671270d7e6e24" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 41189 } } peers { permanent_uuid: "d7d1e2d1cbf944dfb8ec3c087dcd7f1f" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42445 } }
I20260929 18:43:10.616791 18392 heartbeater.cc:503] Master 127.7.157.126:44367 was elected leader, sending a full tablet report...
I20260929 18:43:10.616842 18544 consensus_queue.cc:260] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555cc6035ad646d3a1b7dc0cda437285" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39661 } } peers { permanent_uuid: "37dcbd45cb734188bf4671270d7e6e24" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 41189 } } peers { permanent_uuid: "d7d1e2d1cbf944dfb8ec3c087dcd7f1f" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42445 } }
I20260929 18:43:10.617455 18526 heartbeater.cc:503] Master 127.7.157.126:44367 was elected leader, sending a full tablet report...
I20260929 18:43:10.617690 18544 ts_tablet_manager.cc:1428] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:43:10.618044 18545 ts_tablet_manager.cc:1428] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:43:10.618414 18543 ts_tablet_manager.cc:1428] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:43:10.618677 18258 heartbeater.cc:503] Master 127.7.157.126:44367 was elected leader, sending a full tablet report...
W20260929 18:43:10.624327 18259 tablet.cc:2369] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:43:10.706769 18551 raft_consensus.cc:482] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:43:10.706995 18551 raft_consensus.cc:504] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555cc6035ad646d3a1b7dc0cda437285" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39661 } } peers { permanent_uuid: "37dcbd45cb734188bf4671270d7e6e24" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 41189 } } peers { permanent_uuid: "d7d1e2d1cbf944dfb8ec3c087dcd7f1f" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42445 } }
I20260929 18:43:10.708294 18551 leader_election.cc:288] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 555cc6035ad646d3a1b7dc0cda437285 (127.7.157.65:39661), d7d1e2d1cbf944dfb8ec3c087dcd7f1f (127.7.157.66:42445)
I20260929 18:43:10.712088 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.708551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:39661 (local address 127.7.157.67:56445)
0929 18:43:10.708684 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:10.708701 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:10.708822 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:10.709201 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:10.709204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:10.709222 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:43:10.709332 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:10.709339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.710278 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.710280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:10.711042 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:10.711050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.711681 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.711684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:10.711796 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.711818 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:43:10.711881 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:43:10.711952 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 18:43:10.712087 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.708635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:56445 (local address 127.7.157.65:39661)
0929 18:43:10.708779 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:10.708784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:10.708801 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:43:10.708954 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:10.708957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:10.709026 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:10.709092 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:43:10.709452 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.710181 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.711170 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.711578 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.711683 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.711736 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:10.711908 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 18:43:10.711978 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:43:10.712296 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.708940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:58295 (local address 127.7.157.66:42445)
0929 18:43:10.709066 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:10.709071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:10.709265 (+ 194us) server_negotiation.cc:407] Connection header received
0929 18:43:10.709373 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:10.709376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:10.709426 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:10.709496 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:43:10.709741 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.710401 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.711395 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.711791 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.711901 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.711948 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:10.712104 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 18:43:10.712177 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:43:10.712392 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.708822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:42445 (local address 127.7.157.67:58295)
0929 18:43:10.709169 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:10.709182 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:10.709290 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:10.709509 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:10.709512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:10.709526 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:43:10.709624 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:10.709630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.710507 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.710510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:10.711278 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:10.711286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.711894 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.711898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:10.711993 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.712013 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:43:10.712073 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:43:10.712138 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":117,"threads_started":1}
I20260929 18:43:10.712807 18212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568c16b0936c496eab17a8bc8570f71b" candidate_uuid: "37dcbd45cb734188bf4671270d7e6e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555cc6035ad646d3a1b7dc0cda437285" is_pre_election: true
I20260929 18:43:10.712926 18346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568c16b0936c496eab17a8bc8570f71b" candidate_uuid: "37dcbd45cb734188bf4671270d7e6e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d1e2d1cbf944dfb8ec3c087dcd7f1f" is_pre_election: true
I20260929 18:43:10.713112 18212 raft_consensus.cc:2454] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37dcbd45cb734188bf4671270d7e6e24 in term 0.
I20260929 18:43:10.713202 18346 raft_consensus.cc:2454] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37dcbd45cb734188bf4671270d7e6e24 in term 0.
I20260929 18:43:10.713546 18414 leader_election.cc:302] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dcbd45cb734188bf4671270d7e6e24, 555cc6035ad646d3a1b7dc0cda437285; no voters:
I20260929 18:43:10.713786 18551 raft_consensus.cc:2781] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:43:10.713871 18551 raft_consensus.cc:482] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:43:10.713919 18551 raft_consensus.cc:3037] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:43:10.714938 18551 raft_consensus.cc:504] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555cc6035ad646d3a1b7dc0cda437285" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39661 } } peers { permanent_uuid: "37dcbd45cb734188bf4671270d7e6e24" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 41189 } } peers { permanent_uuid: "d7d1e2d1cbf944dfb8ec3c087dcd7f1f" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42445 } }
I20260929 18:43:10.715269 18551 leader_election.cc:288] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [CANDIDATE]: Term 1 election: Requested vote from peers 555cc6035ad646d3a1b7dc0cda437285 (127.7.157.65:39661), d7d1e2d1cbf944dfb8ec3c087dcd7f1f (127.7.157.66:42445)
I20260929 18:43:10.715658 18346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568c16b0936c496eab17a8bc8570f71b" candidate_uuid: "37dcbd45cb734188bf4671270d7e6e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d1e2d1cbf944dfb8ec3c087dcd7f1f"
I20260929 18:43:10.715829 18346 raft_consensus.cc:3037] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:43:10.716833 18346 raft_consensus.cc:2454] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37dcbd45cb734188bf4671270d7e6e24 in term 1.
I20260929 18:43:10.717075 18212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568c16b0936c496eab17a8bc8570f71b" candidate_uuid: "37dcbd45cb734188bf4671270d7e6e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555cc6035ad646d3a1b7dc0cda437285"
I20260929 18:43:10.717099 18414 leader_election.cc:302] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dcbd45cb734188bf4671270d7e6e24, d7d1e2d1cbf944dfb8ec3c087dcd7f1f; no voters:
I20260929 18:43:10.717195 18212 raft_consensus.cc:3037] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:43:10.717276 18551 raft_consensus.cc:2781] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:43:10.717499 18551 raft_consensus.cc:686] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [term 1 LEADER]: Becoming Leader. State: Replica: 37dcbd45cb734188bf4671270d7e6e24, State: Running, Role: LEADER
I20260929 18:43:10.717864 18551 consensus_queue.cc:237] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555cc6035ad646d3a1b7dc0cda437285" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39661 } } peers { permanent_uuid: "37dcbd45cb734188bf4671270d7e6e24" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 41189 } } peers { permanent_uuid: "d7d1e2d1cbf944dfb8ec3c087dcd7f1f" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42445 } }
I20260929 18:43:10.718304 18212 raft_consensus.cc:2454] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37dcbd45cb734188bf4671270d7e6e24 in term 1.
I20260929 18:43:10.720988 18067 catalog_manager.cc:5495] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37dcbd45cb734188bf4671270d7e6e24 (127.7.157.67). New cstate: current_term: 1 leader_uuid: "37dcbd45cb734188bf4671270d7e6e24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "555cc6035ad646d3a1b7dc0cda437285" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37dcbd45cb734188bf4671270d7e6e24" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d1e2d1cbf944dfb8ec3c087dcd7f1f" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42445 } health_report { overall_health: UNKNOWN } } }
I20260929 18:43:10.747548 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.744096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33940 (local address 127.7.157.67:41189)
0929 18:43:10.744204 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:10.744208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:10.744220 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:43:10.744369 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:10.744372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:10.744427 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:10.744521 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:43:10.744825 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.745314 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.746141 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:10.746349 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:10.746427 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:10.746694 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:43:10.747141 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:43:10.747233 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:10.747365 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 18:43:10.747416 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:43:10.753726 18346 raft_consensus.cc:1261] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f [term 1 FOLLOWER]: Refusing update from remote peer 37dcbd45cb734188bf4671270d7e6e24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:43:10.753726 18212 raft_consensus.cc:1261] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285 [term 1 FOLLOWER]: Refusing update from remote peer 37dcbd45cb734188bf4671270d7e6e24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:43:10.754475 18551 consensus_queue.cc:1035] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d1e2d1cbf944dfb8ec3c087dcd7f1f" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 42445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:43:10.754729 18553 consensus_queue.cc:1035] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555cc6035ad646d3a1b7dc0cda437285" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:43:10.763797 18557 mvcc.cc:205] Tried to move back new op lower bound from 7334737472519180288 to 7334737472383131648. Current Snapshot: MvccSnapshot[applied={T|T < 7334737472519180288}]
I20260929 18:43:10.771538 18064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:43:10.771726 18064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:43:10.774115 18064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aed2d16a37104e7a9d9e492412793c96: Sending DeleteTablet for 3 replicas of tablet 568c16b0936c496eab17a8bc8570f71b
I20260929 18:43:10.774852 18460 tablet_service.cc:1514] Processing DeleteTablet for tablet 568c16b0936c496eab17a8bc8570f71b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:43:10 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:33936
I20260929 18:43:10.774868 18192 tablet_service.cc:1514] Processing DeleteTablet for tablet 568c16b0936c496eab17a8bc8570f71b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:43:10 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:59390
I20260929 18:43:10.775373 18567 tablet_replica.cc:330] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285: stopping tablet replica
I20260929 18:43:10.775375 18326 tablet_service.cc:1514] Processing DeleteTablet for tablet 568c16b0936c496eab17a8bc8570f71b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:43:10 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:34672
I20260929 18:43:10.775698 18567 raft_consensus.cc:2229] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:43:10.775843 18568 tablet_replica.cc:330] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f: stopping tablet replica
I20260929 18:43:10.776008 18567 raft_consensus.cc:2258] T 568c16b0936c496eab17a8bc8570f71b P 555cc6035ad646d3a1b7dc0cda437285 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:43:10.776250 18568 raft_consensus.cc:2229] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:43:10.776418 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 18129
I20260929 18:43:10.777238 18568 raft_consensus.cc:2258] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:43:10.777719 18566 tablet_replica.cc:330] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24: stopping tablet replica
I20260929 18:43:10.777980 18566 raft_consensus.cc:2229] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:43:10.778312 18566 raft_consensus.cc:2258] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:43:10.778766 18568 ts_tablet_manager.cc:1905] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:43:10.779768 18566 ts_tablet_manager.cc:1905] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:43:10.781471 18568 ts_tablet_manager.cc:1918] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:43:10.781567 18568 log.cc:1196] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/568c16b0936c496eab17a8bc8570f71b
I20260929 18:43:10.781883 18568 ts_tablet_manager.cc:1939] T 568c16b0936c496eab17a8bc8570f71b P d7d1e2d1cbf944dfb8ec3c087dcd7f1f: Deleting consensus metadata
I20260929 18:43:10.782450 18566 ts_tablet_manager.cc:1918] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:43:10.782559 18566 log.cc:1196] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/568c16b0936c496eab17a8bc8570f71b
I20260929 18:43:10.782879 18566 ts_tablet_manager.cc:1939] T 568c16b0936c496eab17a8bc8570f71b P 37dcbd45cb734188bf4671270d7e6e24: Deleting consensus metadata
I20260929 18:43:10.783187 18052 catalog_manager.cc:4841] TS d7d1e2d1cbf944dfb8ec3c087dcd7f1f (127.7.157.66:42445): tablet 568c16b0936c496eab17a8bc8570f71b (table test-table [id=a98a7ea88b484f5e805ee968febfe860]) successfully deleted
I20260929 18:43:10.784004 18053 catalog_manager.cc:4841] TS 37dcbd45cb734188bf4671270d7e6e24 (127.7.157.67:41189): tablet 568c16b0936c496eab17a8bc8570f71b (table test-table [id=a98a7ea88b484f5e805ee968febfe860]) successfully deleted
W20260929 18:43:10.784184 18052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:39661 (error 108)
I20260929 18:43:10.784426 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 18263
I20260929 18:43:10.786394 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:10.786081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:39661 (local address 127.0.0.1:59392)
0929 18:43:10.786231 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:10.786307 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:39661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 18:43:10.786535 18052 catalog_manager.cc:4584] TS 555cc6035ad646d3a1b7dc0cda437285 (127.7.157.65:39661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 568c16b0936c496eab17a8bc8570f71b: Network error: Client connection negotiation failed: client connection to 127.7.157.65:39661: connect: Connection refused (error 111)
I20260929 18:43:10.789968 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 18397
I20260929 18:43:10.794946 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 18035
2026-09-29T18:43:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2182 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:43:10 dist-test-slave-smpn krb5kdc[18572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:43:10 dist-test-slave-smpn krb5kdc[18572](info): set up 2 sockets
Sep 29 18:43:10 dist-test-slave-smpn krb5kdc[18572](info): commencing operation
krb5kdc: starting...
W20260929 18:43:11.743947 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:43:11 dist-test-slave-smpn krb5kdc[18572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:43:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:43:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.157.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.157.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.157.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/rangeradmin_127.7.157.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.157.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/rangeradmin_127.7.157.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.157.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.157.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.157.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/rangerlookup_127.7.157.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.157.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/rangerlookup_127.7.157.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.157.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.157.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.157.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/HTTP_127.7.157.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.157.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/HTTP_127.7.157.84@KRBTEST.COM.keytab.
I20260929 18:43:11.908046 7797 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskLJQtbs/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/postgres -l logfile start
2026-09-29 18:43:14.310 UTC [18602] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 18:43:14.311 UTC [18602] LOG: listening on IPv4 address "127.7.157.84", port 52027
2026-09-29 18:43:14.314 UTC [18602] LOG: listening on Unix socket "/tmp/.s.PGSQL.52027"
2026-09-29 18:43:14.334 UTC [18605] LOG: database system was shut down at 2026-09-29 18:43:12 UTC
2026-09-29 18:43:14.338 UTC [18602] LOG: database system is ready to accept connections
I20260929 18:43:15.219657 7797 mini_postgres.cc:96] Postgres bound to 52027
2026-09-29 18:43:15.225 UTC [18613] FATAL: database "slave" does not exist
127.7.157.84:52027 - accepting connections
I20260929 18:43:15.226452 7797 mini_ranger.cc:161] Starting Ranger...
I20260929 18:43:15.241389 7797 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 18:43:15.687086 7797 mini_ranger.cc:87] Created ranger Postgres database
I20260929 18:43:15.687170 7797 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-admin
2026-09-29 18:43:15,948 [I] DB FLAVOR :POSTGRES
2026-09-29 18:43:15,949 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:43:15,949 [I] Checking connection..
2026-09-29 18:43:15,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 18:43:16,210 [I] Checking connection passed.
2026-09-29 18:43:16,210 [I] --------- Verifying version history table ---------
2026-09-29 18:43:16,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:43:16,468 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:43:16,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:43:16,721 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:43:16,721 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 18:43:16,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 18:43:16.974 UTC [18677] WARNING: there is no transaction in progress
2026-09-29 18:43:16,979 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 18:43:16,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:43:17,234 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 18:43:17,234 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:43:17,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 18:43:17,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 18:43:17,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-smpn.c.gcp-upstream.internal','N') ;"
2026-09-29 18:43:17,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:43:18,155 [I] Table x_portal_user does not exist in database ranger
2026-09-29 18:43:18,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:43:18,415 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 18:43:18,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:43:18,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:43:18,930 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 18:43:18,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 18:43:19.887 UTC [18785] WARNING: there is no transaction in progress
2026-09-29 18:43:19.904 UTC [18785] WARNING: there is no transaction in progress
2026-09-29 18:43:19.920 UTC [18785] WARNING: there is no transaction in progress
2026-09-29 18:43:19.935 UTC [18785] WARNING: there is no transaction in progress
2026-09-29 18:43:19.950 UTC [18785] WARNING: there is no transaction in progress
2026-09-29 18:43:19.969 UTC [18785] WARNING: there is no transaction in progress
2026-09-29 18:43:20.034 UTC [18785] WARNING: there is no transaction in progress
2026-09-29 18:43:20.042 UTC [18785] WARNING: there is no transaction in progress
2026-09-29 18:43:20.056 UTC [18785] WARNING: there is no transaction in progress
2026-09-29 18:43:20.067 UTC [18785] WARNING: there is no transaction in progress
2026-09-29 18:43:20.078 UTC [18785] WARNING: there is no transaction in progress
2026-09-29 18:43:20.092 UTC [18785] WARNING: there is no transaction in progress
2026-09-29 18:43:20.101 UTC [18785] WARNING: there is no transaction in progress
2026-09-29 18:43:20.108 UTC [18785] WARNING: there is no transaction in progress
2026-09-29 18:43:20.115 UTC [18785] WARNING: there is no transaction in progress
2026-09-29 18:43:20.122 UTC [18785] WARNING: there is no transaction in progress
2026-09-29 18:43:20,753 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 18:43:20,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 18:43:20,973 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 18:43:20,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:43:21,212 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 18:43:21,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:43:21,473 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 18:43:21,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:43:21,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:43:21,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-smpn.c.gcp-upstream.internal';"
2026-09-29 18:43:22,192 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 18:43:22,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 18:43:22,432 [I] DB_PATCHES have already been applied
I20260929 18:43:22.438232 7797 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-admin:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskLJQtbs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 18:43:22.438305 7797 mini_ranger.cc:193] Using host: 127.7.157.84
I20260929 18:43:22.441605 7797 mini_ranger.cc:227] Ranger admin URL: http://127.7.157.84:55837
Sep 29, 2026 6:43:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:43:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:43:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 6:43:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 6:43:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 6:43:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 6:43:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 18:43:23.446684 7797 mini_ranger.cc:160] Time spent starting Ranger: real 8.220s user 0.004s sys 0.005s
Sep 29, 2026 6:43:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 6:43:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 6:43:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 6:43:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 6:43:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 6:43:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 6:43:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 6:43:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 6:43:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 6:43:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 6:43:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 6:43:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 6:43:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 6:43:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 6:43:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 6:43:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 6:43:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 6:43:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 18:43:45.220114 7797 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.157.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.157.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.157.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/rangerkms_127.7.157.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.157.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/rangerkms_127.7.157.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.157.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.157.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.157.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/HTTP_127.7.157.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.157.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/HTTP_127.7.157.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 18:43:45 dist-test-slave-smpn krb5kdc[18572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707425, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 18:43:45.307921 7797 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 18:43:45.317268 7797 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 18:43:46.135486 7797 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 18:43:46.135605 7797 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-kms
I20260929 18:43:46.135695 7797 mini_ranger_kms.cc:227] Using postgres at 127.7.157.84:52027
2026-09-29 18:43:46,316 [I] DB FLAVOR :POSTGRES
2026-09-29 18:43:46,316 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:43:46,316 [I] Checking connection
2026-09-29 18:43:46,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 18:43:46,559 [I] connection success
2026-09-29 18:43:46,559 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 18:43:46,559 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 18:43:46,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 18:43:46,815 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 18:43:46,815 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:43:46,815 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 18:43:46,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:52027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 18:43:47,067 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 18:43:47.511197 7797 mini_ranger_kms.cc:310] Created kms service
I20260929 18:43:47.683363 7797 mini_ranger_kms.cc:326] Created kudu user
I20260929 18:43:47.724854 7797 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 18:43:47 dist-test-slave-smpn krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707425, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:43:48.112334 7797 mini_ranger_kms.cc:384] Added ranger policy
I20260929 18:43:48.112486 7797 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-kms:/tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 18:43:48.112509 7797 mini_ranger_kms.cc:240] Using host: 127.7.157.84
I20260929 18:43:48.117049 7797 mini_ranger_kms.cc:276] Ranger KMS PID: 18965
I20260929 18:43:48.117182 7797 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.157.84:44373
Sep 29, 2026 6:43:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:43:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:43:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 6:43:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 6:43:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 6:43:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 18:43:50.136669 7797 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.829s user 0.018s sys 0.146s
I20260929 18:43:50.136785 7797 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 18:43:50.136874 7797 mini_ranger_kms.cc:401] 127.7.157.84:44373/kms/v1/keys
Sep 29 18:43:50 dist-test-slave-smpn krb5kdc[18572](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790707430, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.157.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 18:43:50 dist-test-slave-smpn krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707430, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.157.84@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
Sep 29 18:43:51 dist-test-slave-smpn krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707430, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.157.84@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
Sep 29 18:43:53 dist-test-slave-smpn krb5kdc[18572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:43:53.448594 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:36641
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:35043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:36641
--ranger_config_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.157.84:44373/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:43:53.561339 19016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:43:53.561633 19016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:43:53.561672 19016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:43:53.565933 19016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:43:53.566008 19016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:43:53.566026 19016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:43:53.566048 19016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:43:53.566063 19016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:43:53.571817 19016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:35043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.157.84:44373/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:36641
--ranger_config_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:36641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:43:53.572893 19016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:43:53.573964 19016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:43:53.582978 19024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:53.583168 19016 server_base.cc:1034] running on GCE node
W20260929 18:43:53.582998 19022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:53.582978 19021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:53.583877 19016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:43:53.585019 19016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:43:53.586171 19016 hybrid_clock.cc:648] HybridClock initialized: now 1790707433586148 us; error 38 us; skew 500 ppm
Sep 29 18:43:53 dist-test-slave-smpn krb5kdc[18572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707433, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:43:53.593459 19016 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:43:53.594441 19016 webserver.cc:460] Webserver started at http://127.7.157.126:44499/ using document root <none> and password file <none>
I20260929 18:43:53.595001 19016 fs_manager.cc:359] Metadata directory not provided
I20260929 18:43:53.595062 19016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:43:53.595286 19016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:43:53 dist-test-slave-smpn krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707433, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:43:53.644368 19016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "4a5ec80fd74d4ff4b69219c1e8a8b17d"
format_stamp: "Formatted at 2026-09-29 18:43:53 on dist-test-slave-smpn"
server_key: "e2510cf2c164c369f4b7c755c74f1cbb"
server_key_iv: "39b268eb7e19fbd3464001e4f765dee7"
server_key_version: "kuduclusterkey@0"
I20260929 18:43:53.645133 19016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "4a5ec80fd74d4ff4b69219c1e8a8b17d"
format_stamp: "Formatted at 2026-09-29 18:43:53 on dist-test-slave-smpn"
server_key: "e2510cf2c164c369f4b7c755c74f1cbb"
server_key_iv: "39b268eb7e19fbd3464001e4f765dee7"
server_key_version: "kuduclusterkey@0"
I20260929 18:43:53.650301 19016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 18:43:53.668035 19031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:53.669682 19016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:43:53.669821 19016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "4a5ec80fd74d4ff4b69219c1e8a8b17d"
format_stamp: "Formatted at 2026-09-29 18:43:53 on dist-test-slave-smpn"
server_key: "e2510cf2c164c369f4b7c755c74f1cbb"
server_key_iv: "39b268eb7e19fbd3464001e4f765dee7"
server_key_version: "kuduclusterkey@0"
I20260929 18:43:53.669965 19016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:43:53.695593 19016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:43:53.698768 19016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:43:53.698969 19016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:43:53.707760 19016 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:36641
I20260929 18:43:53.707769 19083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:36641 every 8 connection(s)
I20260929 18:43:53.709038 19016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:43:53.711824 19084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:43:53.715986 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 19016
I20260929 18:43:53.716115 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:43:53.716398 7797 external_mini_cluster.cc:1428] Setting key c87b26d8eb4ee943de9ded7fed653691
I20260929 18:43:53.717980 19084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d: Bootstrap starting.
I20260929 18:43:53.721210 19084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d: Neither blocks nor log segments found. Creating new log.
I20260929 18:43:53.722127 19084 log.cc:824] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:43:53.724545 19084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d: No bootstrap required, opened a new log
Sep 29 18:43:53 dist-test-slave-smpn krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:43:53.729110 19084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5ec80fd74d4ff4b69219c1e8a8b17d" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 36641 } }
I20260929 18:43:53.729406 19084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:43:53.729501 19084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a5ec80fd74d4ff4b69219c1e8a8b17d, State: Initialized, Role: FOLLOWER
I20260929 18:43:53.730006 19084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5ec80fd74d4ff4b69219c1e8a8b17d" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 36641 } }
I20260929 18:43:53.730156 19084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:43:53.730239 19084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:43:53.730333 19084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:43:53.731456 19084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5ec80fd74d4ff4b69219c1e8a8b17d" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 36641 } }
I20260929 18:43:53.731825 19084 leader_election.cc:302] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a5ec80fd74d4ff4b69219c1e8a8b17d; no voters:
I20260929 18:43:53.732194 19084 leader_election.cc:288] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:43:53.732293 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:53.718108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50090 (local address 127.7.157.126:36641)
0929 18:43:53.718580 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:53.718600 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:53.718642 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:43:53.719840 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:53.719866 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:53.720319 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:53.720694 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:53.722664 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:53.723800 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:53.724720 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:53.725206 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:53.725352 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:53.727713 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:53.727718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:53.727733 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:43:53.727760 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:53.729240 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:53.729687 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:53.729692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:53.729697 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:53.729828 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:53.730162 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:53.730165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:53.730167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:53.730644 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:53.730832 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:53.731134 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 18:43:53.731431 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":154,"threads_started":1}
I20260929 18:43:53.732287 19089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:43:53.732836 19089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [term 1 LEADER]: Becoming Leader. State: Replica: 4a5ec80fd74d4ff4b69219c1e8a8b17d, State: Running, Role: LEADER
I20260929 18:43:53.733194 19089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5ec80fd74d4ff4b69219c1e8a8b17d" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 36641 } }
I20260929 18:43:53.734344 19089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a5ec80fd74d4ff4b69219c1e8a8b17d. Latest consensus state: current_term: 1 leader_uuid: "4a5ec80fd74d4ff4b69219c1e8a8b17d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5ec80fd74d4ff4b69219c1e8a8b17d" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 36641 } } }
I20260929 18:43:53.734509 19084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:43:53.734521 19089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [sys.catalog]: This master's current role is: LEADER
I20260929 18:43:53.735028 19091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a5ec80fd74d4ff4b69219c1e8a8b17d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a5ec80fd74d4ff4b69219c1e8a8b17d" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 36641 } } }
I20260929 18:43:53.735198 19091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d [sys.catalog]: This master's current role is: LEADER
I20260929 18:43:53.735473 19093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:43:53.738520 19093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:43:53.738927 19084 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:43:53.744415 19093 catalog_manager.cc:1347] Generated new cluster ID: 0626c8db0b4b4129be0df7cfae349a38
I20260929 18:43:53.744580 19093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:43:53.756773 19093 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:43:53.758260 19093 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:43:53.775827 19093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d: Generated new TSK 0
I20260929 18:43:53.776887 19093 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:43:54.489884 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.157.84:44373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:36641
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:35043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:43:54.653834 19129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:43:54.654233 19129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:43:54.654297 19129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:43:54.660539 19129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:43:54.660633 19129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:43:54.660725 19129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:43:54.669147 19129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:35043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.157.84:44373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:36641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:43:54.670696 19129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:43:54.672251 19129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:43:54.700086 19137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:54.700205 19136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:54.700335 19139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:54.700448 19129 server_base.cc:1034] running on GCE node
I20260929 18:43:54.701076 19129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:43:54.701977 19129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:43:54.703154 19129 hybrid_clock.cc:648] HybridClock initialized: now 1790707434703136 us; error 29 us; skew 500 ppm
Sep 29 18:43:54 dist-test-slave-smpn krb5kdc[18572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707434, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:43:54.706657 19129 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:43:54.707960 19129 webserver.cc:460] Webserver started at http://127.7.157.65:33285/ using document root <none> and password file <none>
I20260929 18:43:54.708526 19129 fs_manager.cc:359] Metadata directory not provided
I20260929 18:43:54.708590 19129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:43:54.708782 19129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:43:54 dist-test-slave-smpn krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707434, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:43:54.731998 19129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "4c2041e14f244a2597b3b1c891a5b49b"
format_stamp: "Formatted at 2026-09-29 18:43:54 on dist-test-slave-smpn"
server_key: "af0e4266daaab19cc0c3a7832f6667b6"
server_key_iv: "20e03024f8169461881d214168cdab2c"
server_key_version: "kuduclusterkey@0"
I20260929 18:43:54.732896 19129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "4c2041e14f244a2597b3b1c891a5b49b"
format_stamp: "Formatted at 2026-09-29 18:43:54 on dist-test-slave-smpn"
server_key: "af0e4266daaab19cc0c3a7832f6667b6"
server_key_iv: "20e03024f8169461881d214168cdab2c"
server_key_version: "kuduclusterkey@0"
I20260929 18:43:54.742990 19129 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
I20260929 18:43:54.761868 19147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:54.764062 19129 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:43:54.764271 19129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "4c2041e14f244a2597b3b1c891a5b49b"
format_stamp: "Formatted at 2026-09-29 18:43:54 on dist-test-slave-smpn"
server_key: "af0e4266daaab19cc0c3a7832f6667b6"
server_key_iv: "20e03024f8169461881d214168cdab2c"
server_key_version: "kuduclusterkey@0"
I20260929 18:43:54.764457 19129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:43:54.785162 19129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:43:54.788966 19129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:43:54.789229 19129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:43:54.790055 19129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:43:54.791189 19129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:43:54.791308 19129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:54.791448 19129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:43:54.791538 19129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:54.804237 19129 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:35937
I20260929 18:43:54.805507 19129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:43:54.805591 19261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:35937 every 8 connection(s)
I20260929 18:43:54.809352 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 19129
I20260929 18:43:54.809481 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:43:54.809752 7797 external_mini_cluster.cc:1428] Setting key 8524684cf0809bb6eae98da9054c4d9c
Sep 29 18:43:54 dist-test-slave-smpn krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707434, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:43:54.820041 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:54.807514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:43651 (local address 127.7.157.126:36641)
0929 18:43:54.807777 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:54.807781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:54.808633 (+ 852us) server_negotiation.cc:407] Connection header received
0929 18:43:54.809796 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:54.809799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:54.809872 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:54.809940 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:54.811586 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:54.812327 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:54.813808 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:54.814139 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:54.814233 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:54.816668 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:54.816671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:54.816673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:43:54.816694 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:54.818058 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:54.818556 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:54.818559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:54.818560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:54.818633 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:54.818946 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:54.818949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:54.818951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:54.819174 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:54.819265 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:54.819738 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 18:43:54.819858 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260929 18:43:54.820920 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:54.808000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:36641 (local address 127.7.157.65:43651)
0929 18:43:54.808475 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:54.808527 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:54.809533 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:54.810373 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:54.810382 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:54.810714 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:54.811430 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:54.811450 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:54.812477 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:54.812480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:54.813688 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:54.813697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:54.814277 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:54.814281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:54.814470 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:54.815031 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:54.815047 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:54.816513 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:43:54.818163 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:43:54.818169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:43:54.818185 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:43:54.818453 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:43:54.818713 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:43:54.818716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:43:54.818718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:43:54.818863 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:43:54.819263 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:43:54.819270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:43:54.819513 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 18:43:54.819685 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 18:43:54.820006 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":117,"threads_started":1}
I20260929 18:43:54.822458 19262 heartbeater.cc:347] Connected to a master server at 127.7.157.126:36641
I20260929 18:43:54.822705 19262 heartbeater.cc:464] Registering TS with master...
I20260929 18:43:54.823329 19262 heartbeater.cc:511] Master 127.7.157.126:36641 requested a full tablet report, sending...
I20260929 18:43:54.825099 19048 ts_manager.cc:194] Registered new tserver with Master: 4c2041e14f244a2597b3b1c891a5b49b (127.7.157.65:35937)
I20260929 18:43:54.826196 19048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:43651
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:43:54.851512 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.157.84:44373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:36641
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:35043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:43:54 dist-test-slave-smpn krb5kdc[18572](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790707434, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 18:43:55.003295 19268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:43:55.003746 19268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:43:55.003821 19268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:43:55.010304 19268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:43:55.010462 19268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:43:55.010588 19268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:43:55.019426 19268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:35043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.157.84:44373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:36641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:43:55.020951 19268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:43:55.022478 19268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:43:55.036559 19274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:55.036620 19277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:55.036697 19275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:55.037815 19268 server_base.cc:1034] running on GCE node
I20260929 18:43:55.038362 19268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:43:55.039217 19268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:43:55.040398 19268 hybrid_clock.cc:648] HybridClock initialized: now 1790707435040370 us; error 46 us; skew 500 ppm
Sep 29 18:43:55 dist-test-slave-smpn krb5kdc[18572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707435, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:43:55.044044 19268 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:43:55.045197 19268 webserver.cc:460] Webserver started at http://127.7.157.66:32885/ using document root <none> and password file <none>
I20260929 18:43:55.045918 19268 fs_manager.cc:359] Metadata directory not provided
I20260929 18:43:55.046017 19268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:43:55.046319 19268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:43:55 dist-test-slave-smpn krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707435, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:43:55.069116 19268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "0dbacee01d1e419fa73826c1e86d05ce"
format_stamp: "Formatted at 2026-09-29 18:43:55 on dist-test-slave-smpn"
server_key: "0a2bcd85c04dc4d8289e0c3fb1df7e4e"
server_key_iv: "d1d2b4b3e4c2463bb4a6cd04419a2609"
server_key_version: "kuduclusterkey@0"
I20260929 18:43:55.069998 19268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "0dbacee01d1e419fa73826c1e86d05ce"
format_stamp: "Formatted at 2026-09-29 18:43:55 on dist-test-slave-smpn"
server_key: "0a2bcd85c04dc4d8289e0c3fb1df7e4e"
server_key_iv: "d1d2b4b3e4c2463bb4a6cd04419a2609"
server_key_version: "kuduclusterkey@0"
I20260929 18:43:55.080407 19268 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
I20260929 18:43:55.099438 19284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:55.101313 19268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:43:55.101477 19268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "0dbacee01d1e419fa73826c1e86d05ce"
format_stamp: "Formatted at 2026-09-29 18:43:55 on dist-test-slave-smpn"
server_key: "0a2bcd85c04dc4d8289e0c3fb1df7e4e"
server_key_iv: "d1d2b4b3e4c2463bb4a6cd04419a2609"
server_key_version: "kuduclusterkey@0"
I20260929 18:43:55.101639 19268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:43:55.138748 19268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:43:55.142596 19268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:43:55.142807 19268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:43:55.143622 19268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:43:55.144709 19268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:43:55.144826 19268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:55.144924 19268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:43:55.144990 19268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:55.157740 19268 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:40159
I20260929 18:43:55.157797 19397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:40159 every 8 connection(s)
I20260929 18:43:55.159094 19268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:43:55.163313 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 19268
I20260929 18:43:55.163435 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:43:55.163705 7797 external_mini_cluster.cc:1428] Setting key 2001e7afea67eef202b426159bf55464
Sep 29 18:43:55 dist-test-slave-smpn krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707435, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:43:55.178653 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.162960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:48565 (local address 127.7.157.126:36641)
0929 18:43:55.163133 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:55.163138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:55.164473 (+ 1335us) server_negotiation.cc:407] Connection header received
0929 18:43:55.166053 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:55.166057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:55.166108 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:55.166159 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:55.167909 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.168738 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.170674 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.171022 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.171146 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:55.174194 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:55.174198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:55.174202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:43:55.174229 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:55.176022 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:55.176719 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:55.176723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:55.176725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:55.176813 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:55.177229 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:55.177232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:55.177234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:55.177490 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:55.177572 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:55.178274 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 18:43:55.178452 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:43:55.180675 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.162836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:36641 (local address 127.7.157.66:48565)
0929 18:43:55.164126 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:55.164187 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:55.165796 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:55.166599 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:55.166610 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:55.166981 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:55.167732 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:55.167754 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.168905 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.168909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:55.170539 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:55.170550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.171392 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.171396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:55.171594 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:55.172268 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:55.172287 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:55.173986 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:43:55.176201 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:43:55.176207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:43:55.176226 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:43:55.176600 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:43:55.176922 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:43:55.176925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:43:55.176927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:43:55.177116 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:43:55.177653 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:43:55.177660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:43:55.177994 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 18:43:55.179141 (+ 1147us) client_negotiation.cc:240] Negotiation successful
0929 18:43:55.179529 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"thread_start_us":177,"threads_started":1}
I20260929 18:43:55.182399 19398 heartbeater.cc:347] Connected to a master server at 127.7.157.126:36641
I20260929 18:43:55.182665 19398 heartbeater.cc:464] Registering TS with master...
I20260929 18:43:55.183319 19398 heartbeater.cc:511] Master 127.7.157.126:36641 requested a full tablet report, sending...
I20260929 18:43:55.184651 19048 ts_manager.cc:194] Registered new tserver with Master: 0dbacee01d1e419fa73826c1e86d05ce (127.7.157.66:40159)
I20260929 18:43:55.185417 19048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:48565
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:43:55.209625 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.157.84:44373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:36641
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:35043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:43:55 dist-test-slave-smpn krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707434, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
W20260929 18:43:55.383677 19403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:43:55.384159 19403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:43:55.384254 19403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:43:55.390101 19403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:43:55.390174 19403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:43:55.390255 19403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:43:55.395911 19403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:35043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.157.84:44373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:36641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:43:55.397058 19403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:43:55.398106 19403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:43:55.407742 19408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:55.407889 19411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:43:55.407727 19409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:43:55.408589 19403 server_base.cc:1034] running on GCE node
I20260929 18:43:55.409031 19403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:43:55.409698 19403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:43:55.410872 19403 hybrid_clock.cc:648] HybridClock initialized: now 1790707435410847 us; error 40 us; skew 500 ppm
Sep 29 18:43:55 dist-test-slave-smpn krb5kdc[18572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707435, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:43:55.413838 19403 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:43:55.415045 19403 webserver.cc:460] Webserver started at http://127.7.157.67:46553/ using document root <none> and password file <none>
I20260929 18:43:55.415861 19403 fs_manager.cc:359] Metadata directory not provided
I20260929 18:43:55.415954 19403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:43:55.416198 19403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:43:55 dist-test-slave-smpn krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707435, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:43:55.443382 19403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "749bf2283c31488c8cc3ebee323e18fa"
format_stamp: "Formatted at 2026-09-29 18:43:55 on dist-test-slave-smpn"
server_key: "a9616aa062364b9b9a62859e140e3f90"
server_key_iv: "57a2c807ca02484af99d8a1910ff4e4a"
server_key_version: "kuduclusterkey@0"
I20260929 18:43:55.444182 19403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "749bf2283c31488c8cc3ebee323e18fa"
format_stamp: "Formatted at 2026-09-29 18:43:55 on dist-test-slave-smpn"
server_key: "a9616aa062364b9b9a62859e140e3f90"
server_key_iv: "57a2c807ca02484af99d8a1910ff4e4a"
server_key_version: "kuduclusterkey@0"
I20260929 18:43:55.449221 19403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 18:43:55.475258 19418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:55.477146 19403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:43:55.477371 19403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "749bf2283c31488c8cc3ebee323e18fa"
format_stamp: "Formatted at 2026-09-29 18:43:55 on dist-test-slave-smpn"
server_key: "a9616aa062364b9b9a62859e140e3f90"
server_key_iv: "57a2c807ca02484af99d8a1910ff4e4a"
server_key_version: "kuduclusterkey@0"
I20260929 18:43:55.477618 19403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:43:55.501439 19403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:43:55.505348 19403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:43:55.505596 19403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:43:55.506371 19403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:43:55.507493 19403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:43:55.507612 19403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:55.507709 19403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:43:55.507773 19403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:43:55.518430 19403 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:46593
I20260929 18:43:55.519240 19531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:46593 every 8 connection(s)
I20260929 18:43:55.519759 19403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:43:55.526479 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 19403
I20260929 18:43:55.526613 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:43:55.526880 7797 external_mini_cluster.cc:1428] Setting key 834b408a481c61b1b048afb43e2415ba
Sep 29 18:43:55 dist-test-slave-smpn krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707435, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:43:55.538728 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.523267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:36625 (local address 127.7.157.126:36641)
0929 18:43:55.523399 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:55.523404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:55.525576 (+ 2172us) server_negotiation.cc:407] Connection header received
0929 18:43:55.526859 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:55.526863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:55.526927 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:55.526989 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:55.528726 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.529542 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.531202 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.531547 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.531650 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:55.534400 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:55.534404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:55.534407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:43:55.534435 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:55.536073 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:55.536740 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:55.536744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:55.536746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:55.536842 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:55.537329 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:55.537356 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:55.537358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:55.537628 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:55.537719 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:55.538407 (+ 688us) server_negotiation.cc:299] Negotiation successful
0929 18:43:55.538558 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:43:55.540227 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.522516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:36641 (local address 127.7.157.67:36625)
0929 18:43:55.525378 (+ 2862us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:55.525443 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:55.526584 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:55.527486 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:55.527495 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:55.527856 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:43:55.528552 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:55.528573 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.529695 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.529698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:55.531085 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:55.531097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.531721 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.531726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:55.531930 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:55.532540 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:43:55.532562 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:43:55.534195 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:43:55.536185 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:43:55.536191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:43:55.536209 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:43:55.536623 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:43:55.536943 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:43:55.536946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:43:55.536948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:43:55.537210 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:43:55.537746 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:43:55.537753 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:43:55.538100 (+ 347us) client_negotiation.cc:769] Sending connection context
0929 18:43:55.538324 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 18:43:55.538696 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2622,"thread_start_us":132,"threads_started":1}
I20260929 18:43:55.541720 19532 heartbeater.cc:347] Connected to a master server at 127.7.157.126:36641
I20260929 18:43:55.541958 19532 heartbeater.cc:464] Registering TS with master...
I20260929 18:43:55.542526 19532 heartbeater.cc:511] Master 127.7.157.126:36641 requested a full tablet report, sending...
I20260929 18:43:55.543608 19048 ts_manager.cc:194] Registered new tserver with Master: 749bf2283c31488c8cc3ebee323e18fa (127.7.157.67:46593)
I20260929 18:43:55.544345 19048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:36625
I20260929 18:43:55.552893 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:43:55.565093 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.555724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.7.157.126:36641)
0929 18:43:55.555855 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:55.555859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:55.556417 (+ 558us) server_negotiation.cc:407] Connection header received
0929 18:43:55.556547 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:55.556551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:55.556602 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:55.556688 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:43:55.557542 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.558321 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.559595 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.559919 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.559989 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:55.561698 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:43:55.561703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:43:55.561706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:43:55.561734 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:43:55.563338 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:55.563797 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:55.563800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:55.563802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:55.563881 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:43:55.564143 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:43:55.564146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:43:55.564148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:43:55.564354 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:43:55.564450 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:55.564798 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 18:43:55.564931 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:43:55.571550 19048 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:43:55.576264 19048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 29 18:43:55 dist-test-slave-smpn krb5kdc[18572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707434, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:43:55.605286 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.596109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:40159 (local address 127.0.0.1:56318)
0929 18:43:55.599850 (+ 3741us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:55.599900 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:55.600039 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:55.600611 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:55.600617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:55.600642 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:43:55.600775 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:55.600782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.602206 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.602209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:55.603636 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:55.603647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.604731 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.604735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:55.604848 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:55.604891 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 18:43:55.605012 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 18:43:55.605137 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3571,"thread_start_us":102,"threads_started":1}
I20260929 18:43:55.605290 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.597268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.7.157.66:40159)
0929 18:43:55.599599 (+ 2331us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:55.599612 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:55.600000 (+ 388us) server_negotiation.cc:407] Connection header received
0929 18:43:55.600192 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:55.600198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:55.600432 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:55.600609 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:43:55.600890 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.602088 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.603763 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.604604 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.604709 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:55.604854 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:55.605045 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 18:43:55.605143 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2219,"thread_start_us":100,"threads_started":1}
I20260929 18:43:55.609114 19324 tablet_service.cc:1467] Processing CreateTablet for tablet 5668f5af95b140a8a23a50676acae47c (DEFAULT_TABLE table=test-table [id=d9b641eda3cb4f99a1dfc7bfb61c0128]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:43:55.610325 19324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5668f5af95b140a8a23a50676acae47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:43:55.617697 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.598713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:35937 (local address 127.0.0.1:42994)
0929 18:43:55.605840 (+ 7127us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:55.605853 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:55.605924 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:55.612334 (+ 6410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:55.612337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:55.612353 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:43:55.612419 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:55.612424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.614996 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.614999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:55.615916 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:55.615928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.617292 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.617295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:55.617373 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:55.617397 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:43:55.617468 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 18:43:55.617545 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7073,"thread_start_us":81,"threads_started":1}
I20260929 18:43:55.619343 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.598884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.7.157.65:35937)
0929 18:43:55.610627 (+ 11743us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:55.610633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:55.610659 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:43:55.610743 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:55.610749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:55.610965 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:55.611175 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:43:55.613821 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.614846 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.616036 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.616881 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.616982 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:55.617129 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:55.619047 (+ 1918us) server_negotiation.cc:299] Negotiation successful
0929 18:43:55.619193 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11630,"thread_start_us":98,"threads_started":1}
I20260929 18:43:55.622632 19196 tablet_service.cc:1467] Processing CreateTablet for tablet 5668f5af95b140a8a23a50676acae47c (DEFAULT_TABLE table=test-table [id=d9b641eda3cb4f99a1dfc7bfb61c0128]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:43:55.623845 19196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5668f5af95b140a8a23a50676acae47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:43:55.631702 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.598418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:46593 (local address 127.0.0.1:37006)
0929 18:43:55.600266 (+ 1848us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:55.600278 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:55.600351 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:55.612085 (+ 11734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:55.612089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:55.612108 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:43:55.612221 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:55.612228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.616078 (+ 3850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.616080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:55.617172 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:55.617183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.631218 (+ 14035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.631222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:55.631364 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:55.631395 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 18:43:55.631452 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 18:43:55.631542 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1788,"thread_start_us":94,"threads_started":1}
I20260929 18:43:55.632457 19550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.599116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.7.157.67:46593)
0929 18:43:55.611502 (+ 12386us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:55.611515 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:55.611539 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:43:55.611614 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:55.611619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:55.611775 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:55.611916 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:43:55.612532 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.613673 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.618055 (+ 4382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.618883 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.631929 (+ 13046us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:55.632140 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:55.632199 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 18:43:55.632309 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12273,"thread_start_us":160,"threads_started":1}
I20260929 18:43:55.635107 19552 tablet_bootstrap.cc:492] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b: Bootstrap starting.
I20260929 18:43:55.635689 19466 tablet_service.cc:1467] Processing CreateTablet for tablet 5668f5af95b140a8a23a50676acae47c (DEFAULT_TABLE table=test-table [id=d9b641eda3cb4f99a1dfc7bfb61c0128]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:43:55.637446 19466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5668f5af95b140a8a23a50676acae47c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:43:55.638013 19551 tablet_bootstrap.cc:492] T 5668f5af95b140a8a23a50676acae47c P 0dbacee01d1e419fa73826c1e86d05ce: Bootstrap starting.
I20260929 18:43:55.640477 19551 tablet_bootstrap.cc:654] T 5668f5af95b140a8a23a50676acae47c P 0dbacee01d1e419fa73826c1e86d05ce: Neither blocks nor log segments found. Creating new log.
I20260929 18:43:55.641878 19551 log.cc:824] T 5668f5af95b140a8a23a50676acae47c P 0dbacee01d1e419fa73826c1e86d05ce: Log is configured to *not* fsync() on all Append() calls
I20260929 18:43:55.643199 19552 tablet_bootstrap.cc:654] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b: Neither blocks nor log segments found. Creating new log.
I20260929 18:43:55.644106 19552 log.cc:824] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:43:55.647033 19555 tablet_bootstrap.cc:492] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa: Bootstrap starting.
I20260929 18:43:55.649418 19551 tablet_bootstrap.cc:492] T 5668f5af95b140a8a23a50676acae47c P 0dbacee01d1e419fa73826c1e86d05ce: No bootstrap required, opened a new log
I20260929 18:43:55.649654 19551 ts_tablet_manager.cc:1397] T 5668f5af95b140a8a23a50676acae47c P 0dbacee01d1e419fa73826c1e86d05ce: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260929 18:43:55.650689 19555 tablet_bootstrap.cc:654] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa: Neither blocks nor log segments found. Creating new log.
I20260929 18:43:55.651625 19555 log.cc:824] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa: Log is configured to *not* fsync() on all Append() calls
I20260929 18:43:55.653921 19555 tablet_bootstrap.cc:492] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa: No bootstrap required, opened a new log
I20260929 18:43:55.653829 19551 raft_consensus.cc:348] T 5668f5af95b140a8a23a50676acae47c P 0dbacee01d1e419fa73826c1e86d05ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbacee01d1e419fa73826c1e86d05ce" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 40159 } } peers { permanent_uuid: "749bf2283c31488c8cc3ebee323e18fa" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46593 } } peers { permanent_uuid: "4c2041e14f244a2597b3b1c891a5b49b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 35937 } }
I20260929 18:43:55.654136 19555 ts_tablet_manager.cc:1397] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260929 18:43:55.654152 19551 raft_consensus.cc:374] T 5668f5af95b140a8a23a50676acae47c P 0dbacee01d1e419fa73826c1e86d05ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:43:55.654215 19551 raft_consensus.cc:729] T 5668f5af95b140a8a23a50676acae47c P 0dbacee01d1e419fa73826c1e86d05ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbacee01d1e419fa73826c1e86d05ce, State: Initialized, Role: FOLLOWER
I20260929 18:43:55.654726 19551 consensus_queue.cc:260] T 5668f5af95b140a8a23a50676acae47c P 0dbacee01d1e419fa73826c1e86d05ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbacee01d1e419fa73826c1e86d05ce" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 40159 } } peers { permanent_uuid: "749bf2283c31488c8cc3ebee323e18fa" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46593 } } peers { permanent_uuid: "4c2041e14f244a2597b3b1c891a5b49b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 35937 } }
I20260929 18:43:55.660912 19398 heartbeater.cc:503] Master 127.7.157.126:36641 was elected leader, sending a full tablet report...
I20260929 18:43:55.660997 19551 ts_tablet_manager.cc:1428] T 5668f5af95b140a8a23a50676acae47c P 0dbacee01d1e419fa73826c1e86d05ce: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
W20260929 18:43:55.662026 19399 tablet.cc:2369] T 5668f5af95b140a8a23a50676acae47c P 0dbacee01d1e419fa73826c1e86d05ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:43:55.663381 19552 tablet_bootstrap.cc:492] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b: No bootstrap required, opened a new log
I20260929 18:43:55.663609 19552 ts_tablet_manager.cc:1397] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.002s
I20260929 18:43:55.667373 19555 raft_consensus.cc:348] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbacee01d1e419fa73826c1e86d05ce" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 40159 } } peers { permanent_uuid: "749bf2283c31488c8cc3ebee323e18fa" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46593 } } peers { permanent_uuid: "4c2041e14f244a2597b3b1c891a5b49b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 35937 } }
I20260929 18:43:55.667791 19555 raft_consensus.cc:374] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:43:55.667876 19555 raft_consensus.cc:729] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 749bf2283c31488c8cc3ebee323e18fa, State: Initialized, Role: FOLLOWER
I20260929 18:43:55.668183 19552 raft_consensus.cc:348] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbacee01d1e419fa73826c1e86d05ce" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 40159 } } peers { permanent_uuid: "749bf2283c31488c8cc3ebee323e18fa" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46593 } } peers { permanent_uuid: "4c2041e14f244a2597b3b1c891a5b49b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 35937 } }
I20260929 18:43:55.668453 19552 raft_consensus.cc:374] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:43:55.668517 19552 raft_consensus.cc:729] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2041e14f244a2597b3b1c891a5b49b, State: Initialized, Role: FOLLOWER
I20260929 18:43:55.668548 19555 consensus_queue.cc:260] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbacee01d1e419fa73826c1e86d05ce" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 40159 } } peers { permanent_uuid: "749bf2283c31488c8cc3ebee323e18fa" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46593 } } peers { permanent_uuid: "4c2041e14f244a2597b3b1c891a5b49b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 35937 } }
I20260929 18:43:55.669005 19552 consensus_queue.cc:260] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbacee01d1e419fa73826c1e86d05ce" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 40159 } } peers { permanent_uuid: "749bf2283c31488c8cc3ebee323e18fa" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46593 } } peers { permanent_uuid: "4c2041e14f244a2597b3b1c891a5b49b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 35937 } }
I20260929 18:43:55.670027 19555 ts_tablet_manager.cc:1428] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260929 18:43:55.670091 19552 ts_tablet_manager.cc:1428] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260929 18:43:55.670470 19532 heartbeater.cc:503] Master 127.7.157.126:36641 was elected leader, sending a full tablet report...
I20260929 18:43:55.670580 19262 heartbeater.cc:503] Master 127.7.157.126:36641 was elected leader, sending a full tablet report...
I20260929 18:43:55.723001 19559 raft_consensus.cc:482] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:43:55.723272 19559 raft_consensus.cc:504] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbacee01d1e419fa73826c1e86d05ce" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 40159 } } peers { permanent_uuid: "749bf2283c31488c8cc3ebee323e18fa" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46593 } } peers { permanent_uuid: "4c2041e14f244a2597b3b1c891a5b49b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 35937 } }
I20260929 18:43:55.724709 19559 leader_election.cc:288] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dbacee01d1e419fa73826c1e86d05ce (127.7.157.66:40159), 4c2041e14f244a2597b3b1c891a5b49b (127.7.157.65:35937)
I20260929 18:43:55.730688 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.725161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:40159 (local address 127.7.157.67:47823)
0929 18:43:55.725312 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:55.725331 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:55.725441 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:55.726713 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:55.726717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:55.726739 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:43:55.726862 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:55.726869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.727980 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.727984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:55.729268 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:55.729280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.730255 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.730259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:55.730363 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:55.730388 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:43:55.730459 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 18:43:55.730541 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 18:43:55.732049 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.726022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:47823 (local address 127.7.157.66:40159)
0929 18:43:55.726276 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:55.726281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:55.726296 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:43:55.726350 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:55.726353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:55.726404 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:55.726484 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:43:55.727008 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.727796 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.729412 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.730019 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.730138 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:55.730195 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:55.731830 (+ 1635us) server_negotiation.cc:299] Negotiation successful
0929 18:43:55.731917 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260929 18:43:55.732645 19341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5668f5af95b140a8a23a50676acae47c" candidate_uuid: "749bf2283c31488c8cc3ebee323e18fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbacee01d1e419fa73826c1e86d05ce" is_pre_election: true
I20260929 18:43:55.733004 19341 raft_consensus.cc:2454] T 5668f5af95b140a8a23a50676acae47c P 0dbacee01d1e419fa73826c1e86d05ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 749bf2283c31488c8cc3ebee323e18fa in term 0.
I20260929 18:43:55.733857 19419 leader_election.cc:302] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbacee01d1e419fa73826c1e86d05ce, 749bf2283c31488c8cc3ebee323e18fa; no voters:
I20260929 18:43:55.734319 19559 raft_consensus.cc:2781] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:43:55.734411 19559 raft_consensus.cc:482] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:43:55.734453 19559 raft_consensus.cc:3037] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:43:55.739352 19559 raft_consensus.cc:504] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbacee01d1e419fa73826c1e86d05ce" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 40159 } } peers { permanent_uuid: "749bf2283c31488c8cc3ebee323e18fa" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46593 } } peers { permanent_uuid: "4c2041e14f244a2597b3b1c891a5b49b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 35937 } }
I20260929 18:43:55.741070 19341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5668f5af95b140a8a23a50676acae47c" candidate_uuid: "749bf2283c31488c8cc3ebee323e18fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbacee01d1e419fa73826c1e86d05ce"
I20260929 18:43:55.741232 19341 raft_consensus.cc:3037] T 5668f5af95b140a8a23a50676acae47c P 0dbacee01d1e419fa73826c1e86d05ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:43:55.742699 19341 raft_consensus.cc:2454] T 5668f5af95b140a8a23a50676acae47c P 0dbacee01d1e419fa73826c1e86d05ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 749bf2283c31488c8cc3ebee323e18fa in term 1.
I20260929 18:43:55.743227 19419 leader_election.cc:302] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbacee01d1e419fa73826c1e86d05ce, 749bf2283c31488c8cc3ebee323e18fa; no voters:
I20260929 18:43:55.745039 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.725423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:35937 (local address 127.7.157.67:34707)
0929 18:43:55.730859 (+ 5436us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:55.730874 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:55.730951 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:55.736378 (+ 5427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:43:55.736383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:43:55.736415 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:43:55.736536 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:55.736544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.738095 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.738100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:55.740155 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:43:55.740169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.744586 (+ 4417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.744590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:43:55.744687 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:55.744711 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:43:55.744773 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:43:55.744858 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5382,"thread_start_us":115,"threads_started":1}
I20260929 18:43:55.745494 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.726153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:34707 (local address 127.7.157.65:35937)
0929 18:43:55.726655 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:55.726660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:55.731249 (+ 4589us) server_negotiation.cc:407] Connection header received
0929 18:43:55.731312 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:55.731315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:55.731371 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:55.731443 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:43:55.736713 (+ 5270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.737600 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.743642 (+ 6042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.744311 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.744425 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:55.744482 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:55.745290 (+ 808us) server_negotiation.cc:299] Negotiation successful
0929 18:43:55.745375 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418}
I20260929 18:43:55.746865 19205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5668f5af95b140a8a23a50676acae47c" candidate_uuid: "749bf2283c31488c8cc3ebee323e18fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2041e14f244a2597b3b1c891a5b49b" is_pre_election: true
I20260929 18:43:55.747228 19205 raft_consensus.cc:2454] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 749bf2283c31488c8cc3ebee323e18fa in term 0.
I20260929 18:43:55.747256 19562 raft_consensus.cc:2781] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:43:55.747499 19562 raft_consensus.cc:686] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [term 1 LEADER]: Becoming Leader. State: Replica: 749bf2283c31488c8cc3ebee323e18fa, State: Running, Role: LEADER
I20260929 18:43:55.747583 19206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5668f5af95b140a8a23a50676acae47c" candidate_uuid: "749bf2283c31488c8cc3ebee323e18fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2041e14f244a2597b3b1c891a5b49b"
I20260929 18:43:55.747675 19206 raft_consensus.cc:3037] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:43:55.747879 19562 consensus_queue.cc:237] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbacee01d1e419fa73826c1e86d05ce" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 40159 } } peers { permanent_uuid: "749bf2283c31488c8cc3ebee323e18fa" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46593 } } peers { permanent_uuid: "4c2041e14f244a2597b3b1c891a5b49b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 35937 } }
I20260929 18:43:55.748927 19206 raft_consensus.cc:2454] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 749bf2283c31488c8cc3ebee323e18fa in term 1.
I20260929 18:43:55.745939 19559 leader_election.cc:288] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [CANDIDATE]: Term 1 election: Requested vote from peers 0dbacee01d1e419fa73826c1e86d05ce (127.7.157.66:40159), 4c2041e14f244a2597b3b1c891a5b49b (127.7.157.65:35937)
I20260929 18:43:55.754631 19047 catalog_manager.cc:5495] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 749bf2283c31488c8cc3ebee323e18fa (127.7.157.67). New cstate: current_term: 1 leader_uuid: "749bf2283c31488c8cc3ebee323e18fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbacee01d1e419fa73826c1e86d05ce" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 40159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "749bf2283c31488c8cc3ebee323e18fa" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 46593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c2041e14f244a2597b3b1c891a5b49b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 35937 } health_report { overall_health: UNKNOWN } } }
W20260929 18:43:55.772050 19533 tablet.cc:2369] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:43:55.803793 19550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.798372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.7.157.67:46593)
0929 18:43:55.798501 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:43:55.798506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:43:55.798786 (+ 280us) server_negotiation.cc:407] Connection header received
0929 18:43:55.798908 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:43:55.798911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:43:55.798991 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:43:55.799181 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:43:55.799631 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.800382 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.801689 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:43:55.802000 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:43:55.802133 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:43:55.802633 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:43:55.803250 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:43:55.803364 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:43:55.803546 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 18:43:55.803621 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 18:43:55.808058 19263 tablet.cc:2369] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:43:55.812235 19206 raft_consensus.cc:1261] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b [term 1 FOLLOWER]: Refusing update from remote peer 749bf2283c31488c8cc3ebee323e18fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:43:55.813133 19563 consensus_queue.cc:1035] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c2041e14f244a2597b3b1c891a5b49b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 35937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:43:55.820869 19341 raft_consensus.cc:1261] T 5668f5af95b140a8a23a50676acae47c P 0dbacee01d1e419fa73826c1e86d05ce [term 1 FOLLOWER]: Refusing update from remote peer 749bf2283c31488c8cc3ebee323e18fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:43:55.821686 19563 consensus_queue.cc:1035] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dbacee01d1e419fa73826c1e86d05ce" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:43:55.836676 19566 mvcc.cc:205] Tried to move back new op lower bound from 7334737657077334016 to 7334737656838488064. Current Snapshot: MvccSnapshot[applied={T|T < 7334737657077334016}]
I20260929 18:43:55.838730 19569 mvcc.cc:205] Tried to move back new op lower bound from 7334737657077334016 to 7334737656838488064. Current Snapshot: MvccSnapshot[applied={T|T < 7334737657077334016}]
I20260929 18:43:55.846529 19567 mvcc.cc:205] Tried to move back new op lower bound from 7334737657077334016 to 7334737656838488064. Current Snapshot: MvccSnapshot[applied={T|T < 7334737657077334016}]
I20260929 18:43:55.852537 19047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:43:55.852735 19047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:43:55.856030 19047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a5ec80fd74d4ff4b69219c1e8a8b17d: Sending DeleteTablet for 3 replicas of tablet 5668f5af95b140a8a23a50676acae47c
I20260929 18:43:55.857372 19324 tablet_service.cc:1514] Processing DeleteTablet for tablet 5668f5af95b140a8a23a50676acae47c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:43:55 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:56318
I20260929 18:43:55.857498 19196 tablet_service.cc:1514] Processing DeleteTablet for tablet 5668f5af95b140a8a23a50676acae47c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:43:55 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:42994
I20260929 18:43:55.858718 19576 tablet_replica.cc:330] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b: stopping tablet replica
I20260929 18:43:55.858692 19466 tablet_service.cc:1514] Processing DeleteTablet for tablet 5668f5af95b140a8a23a50676acae47c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:43:55 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:37006
I20260929 18:43:55.858925 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 19129
I20260929 18:43:55.859205 19576 raft_consensus.cc:2229] T 5668f5af95b140a8a23a50676acae47c P 4c2041e14f244a2597b3b1c891a5b49b [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 18:43:55.864180 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:35937 (error 108)
I20260929 18:43:55.864763 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 19268
I20260929 18:43:55.871352 19577 tablet_replica.cc:330] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa: stopping tablet replica
I20260929 18:43:55.871752 19577 raft_consensus.cc:2229] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:43:55.872279 19577 raft_consensus.cc:2258] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:43:55.876369 19577 ts_tablet_manager.cc:1905] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:43:55.884393 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.884057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:35937 (local address 127.0.0.1:47718)
0929 18:43:55.884204 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:55.884287 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 18:43:55.884681 19577 ts_tablet_manager.cc:1918] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260929 18:43:55.884743 19032 catalog_manager.cc:4584] TS 4c2041e14f244a2597b3b1c891a5b49b (127.7.157.65:35937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5668f5af95b140a8a23a50676acae47c: Network error: Client connection negotiation failed: client connection to 127.7.157.65:35937: connect: Connection refused (error 111)
I20260929 18:43:55.884802 19577 log.cc:1196] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/5668f5af95b140a8a23a50676acae47c
I20260929 18:43:55.885154 19577 ts_tablet_manager.cc:1939] T 5668f5af95b140a8a23a50676acae47c P 749bf2283c31488c8cc3ebee323e18fa: Deleting consensus metadata
I20260929 18:43:55.886500 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:43:55.886206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:40159 (local address 127.0.0.1:39234)
0929 18:43:55.886367 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:55.886418 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.66:40159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 18:43:55.886912 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 19403
W20260929 18:43:55.896464 19545 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:43:55.895734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:46593 (local address 127.0.0.1:54202)
0929 18:43:55.895920 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 18:43:55.895935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:43:55.896027 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:43:55.896333 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.67:46593: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 18:43:55.897207 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 19016
I20260929 18:43:55.904472 7797 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 18:43:56.249560 7797 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 18:43:56.249711 7797 mini_ranger.cc:66] Stopping Ranger...
I20260929 18:43:56.583535 7797 mini_ranger.cc:68] Stopped Ranger
2026-09-29 18:43:56.583 UTC [18602] LOG: received smart shutdown request
2026-09-29 18:43:56.587 UTC [18602] LOG: background worker "logical replication launcher" (PID 18611) exited with exit code 1
2026-09-29 18:43:56.587 UTC [18606] LOG: shutting down
2026-09-29 18:43:56.742 UTC [18602] LOG: database system is shut down
2026-09-29T18:43:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46035 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:43:56 dist-test-slave-smpn krb5kdc[19587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:43:56 dist-test-slave-smpn krb5kdc[19587](info): set up 2 sockets
Sep 29 18:43:56 dist-test-slave-smpn krb5kdc[19587](info): commencing operation
krb5kdc: starting...
W20260929 18:43:57.778863 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:43:57 dist-test-slave-smpn krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:43:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:43:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.157.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.157.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.157.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/rangeradmin_127.7.157.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.157.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/rangeradmin_127.7.157.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.157.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.157.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.157.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/rangerlookup_127.7.157.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.157.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/rangerlookup_127.7.157.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.157.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.157.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.157.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/HTTP_127.7.157.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.157.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/HTTP_127.7.157.84@KRBTEST.COM.keytab.
I20260929 18:43:57.934235 7797 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskLJQtbs/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/postgres -l logfile start
2026-09-29 18:44:00.216 UTC [19617] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 18:44:00.216 UTC [19617] LOG: listening on IPv4 address "127.7.157.84", port 41805
2026-09-29 18:44:00.219 UTC [19617] LOG: listening on Unix socket "/tmp/.s.PGSQL.41805"
2026-09-29 18:44:00.238 UTC [19620] LOG: database system was shut down at 2026-09-29 18:43:58 UTC
2026-09-29 18:44:00.242 UTC [19617] LOG: database system is ready to accept connections
I20260929 18:44:01.110697 7797 mini_postgres.cc:96] Postgres bound to 41805
2026-09-29 18:44:01.116 UTC [19628] FATAL: database "slave" does not exist
127.7.157.84:41805 - accepting connections
I20260929 18:44:01.117229 7797 mini_ranger.cc:161] Starting Ranger...
I20260929 18:44:01.131477 7797 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 18:44:01.560504 7797 mini_ranger.cc:87] Created ranger Postgres database
I20260929 18:44:01.560593 7797 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-admin
2026-09-29 18:44:01,849 [I] DB FLAVOR :POSTGRES
2026-09-29 18:44:01,849 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:44:01,849 [I] Checking connection..
2026-09-29 18:44:01,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 18:44:02,101 [I] Checking connection passed.
2026-09-29 18:44:02,101 [I] --------- Verifying version history table ---------
2026-09-29 18:44:02,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:44:02,356 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:44:02,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:44:02,597 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:44:02,597 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 18:44:02,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 18:44:02.850 UTC [19692] WARNING: there is no transaction in progress
2026-09-29 18:44:02,856 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 18:44:02,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:44:03,093 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 18:44:03,093 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:44:03,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 18:44:03,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 18:44:03,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-smpn.c.gcp-upstream.internal','N') ;"
2026-09-29 18:44:03,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:44:04,034 [I] Table x_portal_user does not exist in database ranger
2026-09-29 18:44:04,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:44:04,271 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 18:44:04,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:44:04,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:44:04,754 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 18:44:04,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 18:44:05.681 UTC [19800] WARNING: there is no transaction in progress
2026-09-29 18:44:05.696 UTC [19800] WARNING: there is no transaction in progress
2026-09-29 18:44:05.711 UTC [19800] WARNING: there is no transaction in progress
2026-09-29 18:44:05.725 UTC [19800] WARNING: there is no transaction in progress
2026-09-29 18:44:05.740 UTC [19800] WARNING: there is no transaction in progress
2026-09-29 18:44:05.757 UTC [19800] WARNING: there is no transaction in progress
2026-09-29 18:44:05.811 UTC [19800] WARNING: there is no transaction in progress
2026-09-29 18:44:05.817 UTC [19800] WARNING: there is no transaction in progress
2026-09-29 18:44:05.831 UTC [19800] WARNING: there is no transaction in progress
2026-09-29 18:44:05.840 UTC [19800] WARNING: there is no transaction in progress
2026-09-29 18:44:05.851 UTC [19800] WARNING: there is no transaction in progress
2026-09-29 18:44:05.864 UTC [19800] WARNING: there is no transaction in progress
2026-09-29 18:44:05.873 UTC [19800] WARNING: there is no transaction in progress
2026-09-29 18:44:05.879 UTC [19800] WARNING: there is no transaction in progress
2026-09-29 18:44:05.885 UTC [19800] WARNING: there is no transaction in progress
2026-09-29 18:44:05.891 UTC [19800] WARNING: there is no transaction in progress
2026-09-29 18:44:06,476 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 18:44:06,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 18:44:06,717 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 18:44:06,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:44:06,975 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 18:44:06,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:44:07,208 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 18:44:07,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:44:07,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:44:07,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-smpn.c.gcp-upstream.internal';"
2026-09-29 18:44:07,951 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 18:44:07,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 18:44:08,182 [I] DB_PATCHES have already been applied
I20260929 18:44:08.187300 7797 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-admin:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskLJQtbs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 18:44:08.187363 7797 mini_ranger.cc:193] Using host: 127.7.157.84
I20260929 18:44:08.190637 7797 mini_ranger.cc:227] Ranger admin URL: http://127.7.157.84:35465
Sep 29, 2026 6:44:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:44:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:44:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 6:44:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 6:44:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 6:44:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 6:44:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 18:44:09.175532 7797 mini_ranger.cc:160] Time spent starting Ranger: real 8.058s user 0.000s sys 0.009s
Sep 29, 2026 6:44:29 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 6:44:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 6:44:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 6:44:29 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 6:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 6:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 6:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 6:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 6:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 6:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 6:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 6:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 6:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 6:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 6:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 6:44:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 6:44:29 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 6:44:30 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 18:44:30.773823 7797 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.157.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.157.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.157.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/rangerkms_127.7.157.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.157.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/rangerkms_127.7.157.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.157.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.157.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.157.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/HTTP_127.7.157.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.157.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/HTTP_127.7.157.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 18:44:30 dist-test-slave-smpn krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707470, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 18:44:30.852878 7797 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 18:44:30.862293 7797 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 18:44:31.642521 7797 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 18:44:31.642593 7797 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-kms
I20260929 18:44:31.642611 7797 mini_ranger_kms.cc:227] Using postgres at 127.7.157.84:41805
2026-09-29 18:44:31,794 [I] DB FLAVOR :POSTGRES
2026-09-29 18:44:31,794 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:44:31,794 [I] Checking connection
2026-09-29 18:44:31,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 18:44:32,029 [I] connection success
2026-09-29 18:44:32,030 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 18:44:32,030 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 18:44:32,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 18:44:32,259 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 18:44:32,260 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:44:32,260 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 18:44:32,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.157.84:41805/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 18:44:32,509 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 18:44:32.990315 7797 mini_ranger_kms.cc:310] Created kms service
I20260929 18:44:33.217181 7797 mini_ranger_kms.cc:326] Created kudu user
I20260929 18:44:33.259483 7797 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 18:44:33 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707470, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:44:33.638825 7797 mini_ranger_kms.cc:384] Added ranger policy
I20260929 18:44:33.638967 7797 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-kms:/tmp/dist-test-taskLJQtbs/build/debug/bin/postgresql.jar:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskLJQtbs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 18:44:33.638988 7797 mini_ranger_kms.cc:240] Using host: 127.7.157.84
I20260929 18:44:33.642771 7797 mini_ranger_kms.cc:276] Ranger KMS PID: 19980
I20260929 18:44:33.642838 7797 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.157.84:45107
Sep 29, 2026 6:44:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:44:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:44:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 6:44:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 6:44:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 6:44:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 18:44:35.662168 7797 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.809s user 0.008s sys 0.128s
I20260929 18:44:35.662276 7797 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 18:44:35.662336 7797 mini_ranger_kms.cc:401] 127.7.157.84:45107/kms/v1/keys
Sep 29 18:44:35 dist-test-slave-smpn krb5kdc[19587](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790707475, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.157.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 18:44:36 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707475, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.157.84@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
Sep 29 18:44:36 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707475, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.157.84@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
Sep 29 18:44:38 dist-test-slave-smpn krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:44:38.926733 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:40455
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:40455
--ranger_config_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.157.84:45107/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:44:39.039297 20031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:39.039666 20031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:39.039777 20031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:39.044034 20031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:44:39.044107 20031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:39.044139 20031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:39.044173 20031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:44:39.044209 20031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:44:39.049713 20031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.157.84:45107/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:40455
--ranger_config_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:40455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:39.050884 20031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:39.052023 20031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:44:39.061167 20031 server_base.cc:1034] running on GCE node
W20260929 18:44:39.061116 20036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:39.061158 20037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:39.061116 20039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:39.062007 20031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:39.063211 20031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:39.064375 20031 hybrid_clock.cc:648] HybridClock initialized: now 1790707479064363 us; error 35 us; skew 500 ppm
Sep 29 18:44:39 dist-test-slave-smpn krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707479, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:39.067129 20031 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:44:39.068029 20031 webserver.cc:460] Webserver started at http://127.7.157.126:38407/ using document root <none> and password file <none>
I20260929 18:44:39.068585 20031 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:39.068650 20031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:39.068846 20031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:44:39 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707479, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:44:39.118086 20031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "944ac7088b084cf79e140025059c49fa"
format_stamp: "Formatted at 2026-09-29 18:44:39 on dist-test-slave-smpn"
server_key: "0f52ae0e11a1beb674c799ffbb7fe26a"
server_key_iv: "e4fd347324d3ecd7f835926084b930e3"
server_key_version: "kuduclusterkey@0"
I20260929 18:44:39.118886 20031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "944ac7088b084cf79e140025059c49fa"
format_stamp: "Formatted at 2026-09-29 18:44:39 on dist-test-slave-smpn"
server_key: "0f52ae0e11a1beb674c799ffbb7fe26a"
server_key_iv: "e4fd347324d3ecd7f835926084b930e3"
server_key_version: "kuduclusterkey@0"
I20260929 18:44:39.124121 20031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260929 18:44:39.141736 20046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:39.143471 20031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:44:39.143631 20031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "944ac7088b084cf79e140025059c49fa"
format_stamp: "Formatted at 2026-09-29 18:44:39 on dist-test-slave-smpn"
server_key: "0f52ae0e11a1beb674c799ffbb7fe26a"
server_key_iv: "e4fd347324d3ecd7f835926084b930e3"
server_key_version: "kuduclusterkey@0"
I20260929 18:44:39.143800 20031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:39.165997 20031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:39.169502 20031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:39.169724 20031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:39.178194 20098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:40455 every 8 connection(s)
I20260929 18:44:39.178210 20031 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:40455
I20260929 18:44:39.179486 20031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:44:39.182267 20099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:44:39.184515 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 20031
I20260929 18:44:39.184641 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:44:39.184944 7797 external_mini_cluster.cc:1428] Setting key 257884243b8b949c5eedb3d59155c840
I20260929 18:44:39.188321 20099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa: Bootstrap starting.
I20260929 18:44:39.190796 20099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa: Neither blocks nor log segments found. Creating new log.
I20260929 18:44:39.191574 20099 log.cc:824] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:39.194087 20099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa: No bootstrap required, opened a new log
Sep 29 18:44:39 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:39.197962 20099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944ac7088b084cf79e140025059c49fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40455 } }
I20260929 18:44:39.198287 20099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:44:39.198378 20099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944ac7088b084cf79e140025059c49fa, State: Initialized, Role: FOLLOWER
I20260929 18:44:39.198935 20099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944ac7088b084cf79e140025059c49fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40455 } }
I20260929 18:44:39.199134 20099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:44:39.199225 20099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:44:39.199325 20099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:44:39.200446 20099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944ac7088b084cf79e140025059c49fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40455 } }
I20260929 18:44:39.200841 20099 leader_election.cc:302] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 944ac7088b084cf79e140025059c49fa; no voters:
I20260929 18:44:39.201181 20099 leader_election.cc:288] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:44:39.201382 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:39.186393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41446 (local address 127.7.157.126:40455)
0929 18:44:39.186844 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:39.186862 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:39.186904 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:44:39.187748 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:39.187771 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:39.188170 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:39.188515 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:39.189451 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:39.190868 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:39.192199 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:39.192652 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:39.192770 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:39.195780 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:39.195787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:39.195805 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:39.195839 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:39.197826 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:39.198362 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:39.198369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:39.198375 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:39.198517 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:39.198846 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:39.198849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:39.198851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:39.199435 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:39.199705 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:39.200048 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 18:44:39.200345 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":135,"threads_started":1}
I20260929 18:44:39.202407 20099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:44:39.201390 20104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:44:39.202752 20104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 1 LEADER]: Becoming Leader. State: Replica: 944ac7088b084cf79e140025059c49fa, State: Running, Role: LEADER
I20260929 18:44:39.203099 20104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944ac7088b084cf79e140025059c49fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40455 } }
I20260929 18:44:39.204813 20099 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:44:39.204912 20105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "944ac7088b084cf79e140025059c49fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944ac7088b084cf79e140025059c49fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40455 } } }
I20260929 18:44:39.204937 20106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 944ac7088b084cf79e140025059c49fa. Latest consensus state: current_term: 1 leader_uuid: "944ac7088b084cf79e140025059c49fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944ac7088b084cf79e140025059c49fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40455 } } }
I20260929 18:44:39.205054 20105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [sys.catalog]: This master's current role is: LEADER
I20260929 18:44:39.205055 20106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [sys.catalog]: This master's current role is: LEADER
I20260929 18:44:39.205422 20118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:44:39.210434 20118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:44:39.216650 20118 catalog_manager.cc:1347] Generated new cluster ID: 070860578bfc494695cfd843fdcdeecc
I20260929 18:44:39.216729 20118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:44:39.228608 20118 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:44:39.229888 20118 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:44:39.237869 20118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa: Generated new TSK 0
I20260929 18:44:39.238720 20118 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:44:39.902241 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.157.84:45107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:40455
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:44:40.041250 20144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:40.041641 20144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:40.041728 20144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:40.046896 20144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:40.046965 20144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:40.047031 20144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:44:40.052855 20144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.157.84:45107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:40455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:40.053982 20144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:40.055096 20144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:40.066958 20154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:40.068328 20144 server_base.cc:1034] running on GCE node
W20260929 18:44:40.068934 20151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:40.069105 20152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:40.069705 20144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:40.070493 20144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:40.073755 20144 hybrid_clock.cc:648] HybridClock initialized: now 1790707480073679 us; error 86 us; skew 500 ppm
Sep 29 18:44:40 dist-test-slave-smpn krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:40.081465 20144 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:44:40.082729 20144 webserver.cc:460] Webserver started at http://127.7.157.65:45865/ using document root <none> and password file <none>
I20260929 18:44:40.083575 20144 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:40.083719 20144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:40.084030 20144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:44:40 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:44:40.101289 20144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "79f787b4b1574c6eb68f0183f6112890"
format_stamp: "Formatted at 2026-09-29 18:44:40 on dist-test-slave-smpn"
server_key: "754592c54464246e7646887f97d23481"
server_key_iv: "83c2b264819c1c5a29515169f035aa79"
server_key_version: "kuduclusterkey@0"
I20260929 18:44:40.102082 20144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "79f787b4b1574c6eb68f0183f6112890"
format_stamp: "Formatted at 2026-09-29 18:44:40 on dist-test-slave-smpn"
server_key: "754592c54464246e7646887f97d23481"
server_key_iv: "83c2b264819c1c5a29515169f035aa79"
server_key_version: "kuduclusterkey@0"
I20260929 18:44:40.106734 20144 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:44:40.126713 20163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:40.128638 20144 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260929 18:44:40.128801 20144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "79f787b4b1574c6eb68f0183f6112890"
format_stamp: "Formatted at 2026-09-29 18:44:40 on dist-test-slave-smpn"
server_key: "754592c54464246e7646887f97d23481"
server_key_iv: "83c2b264819c1c5a29515169f035aa79"
server_key_version: "kuduclusterkey@0"
I20260929 18:44:40.128952 20144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:40.159396 20144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:40.162943 20144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:40.163251 20144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:40.164047 20144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:44:40.165159 20144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:44:40.165271 20144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:40.165412 20144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:44:40.165485 20144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:40.178879 20144 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:33095
Sep 29 18:44:40 dist-test-slave-smpn krb5kdc[19587](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790707480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:40.180178 20144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:44:40.180783 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 20144
I20260929 18:44:40.180877 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:44:40.181180 7797 external_mini_cluster.cc:1428] Setting key 5f6fb8ef6e4e0e445c6ca255bdf81eab
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
I20260929 18:44:40.191777 20276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:33095 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
Sep 29 18:44:40 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:40.214506 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:40.195537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:39507 (local address 127.7.157.126:40455)
0929 18:44:40.201070 (+ 5533us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:40.201077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:40.201090 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:44:40.201126 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:40.201128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:40.201169 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:40.201245 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:40.202848 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:40.203675 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:40.205360 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:40.205724 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:40.205806 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:40.209274 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:40.209279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:40.209281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:40.209311 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:40.211432 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:40.212561 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:40.212565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:40.212567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:40.212669 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:40.213134 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:40.213138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:40.213139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:40.213361 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:40.213455 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:40.214180 (+ 725us) server_negotiation.cc:299] Negotiation successful
0929 18:44:40.214322 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5448,"thread_start_us":103,"threads_started":1}
I20260929 18:44:40.215790 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:40.194359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:40455 (local address 127.7.157.65:39507)
0929 18:44:40.199465 (+ 5106us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:40.199537 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:40.200665 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:40.201517 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:40.201528 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:40.201932 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:40.202658 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:40.202680 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:40.203832 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:40.203835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:40.205223 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:40.205235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:40.206039 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:40.206044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:40.206358 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:40.207208 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:40.207230 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:40.209041 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:40.211589 (+ 2548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:40.211596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:40.211616 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:40.212017 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:40.212782 (+ 765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:40.212786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:40.212789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:40.213015 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:40.213476 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:40.213483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:40.213836 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 18:44:40.214095 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 18:44:40.214475 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4831,"thread_start_us":140,"threads_started":1}
I20260929 18:44:40.217629 20277 heartbeater.cc:347] Connected to a master server at 127.7.157.126:40455
I20260929 18:44:40.217886 20277 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:40.218477 20277 heartbeater.cc:511] Master 127.7.157.126:40455 requested a full tablet report, sending...
I20260929 18:44:40.220330 20063 ts_manager.cc:194] Registered new tserver with Master: 79f787b4b1574c6eb68f0183f6112890 (127.7.157.65:33095)
I20260929 18:44:40.221956 20063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:39507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:44:40.223039 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.157.84:45107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:40455
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:44:40.382978 20283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:40.383445 20283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:40.383554 20283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:40.389642 20283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:40.389804 20283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:40.389931 20283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:44:40.397846 20283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.157.84:45107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:40455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:40.399519 20283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:40.400971 20283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:40.411988 20289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:40.411952 20290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:40.412613 20292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:40.413152 20283 server_base.cc:1034] running on GCE node
I20260929 18:44:40.413769 20283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:40.414593 20283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:40.415781 20283 hybrid_clock.cc:648] HybridClock initialized: now 1790707480415754 us; error 39 us; skew 500 ppm
Sep 29 18:44:40 dist-test-slave-smpn krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:40.418707 20283 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:44:40.419657 20283 webserver.cc:460] Webserver started at http://127.7.157.66:38099/ using document root <none> and password file <none>
I20260929 18:44:40.420223 20283 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:40.420293 20283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:40.420500 20283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:44:40 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:44:40.435544 20283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "ceb7932d7a32449cb24bfdde7f67d33d"
format_stamp: "Formatted at 2026-09-29 18:44:40 on dist-test-slave-smpn"
server_key: "dd2639c86418a51b0c4f04ede2d21543"
server_key_iv: "ca64d4f11a30e23e748dcb2090397e18"
server_key_version: "kuduclusterkey@0"
I20260929 18:44:40.436357 20283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "ceb7932d7a32449cb24bfdde7f67d33d"
format_stamp: "Formatted at 2026-09-29 18:44:40 on dist-test-slave-smpn"
server_key: "dd2639c86418a51b0c4f04ede2d21543"
server_key_iv: "ca64d4f11a30e23e748dcb2090397e18"
server_key_version: "kuduclusterkey@0"
I20260929 18:44:40.446713 20283 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
I20260929 18:44:40.479534 20299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:40.481551 20283 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.002s sys 0.001s
I20260929 18:44:40.481812 20283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "ceb7932d7a32449cb24bfdde7f67d33d"
format_stamp: "Formatted at 2026-09-29 18:44:40 on dist-test-slave-smpn"
server_key: "dd2639c86418a51b0c4f04ede2d21543"
server_key_iv: "ca64d4f11a30e23e748dcb2090397e18"
server_key_version: "kuduclusterkey@0"
I20260929 18:44:40.482067 20283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:40.498801 20283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:40.503798 20283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:40.504125 20283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:40.504879 20283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:44:40.506685 20283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:44:40.506752 20283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:40.506816 20283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:44:40.506843 20283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:40.518540 20283 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:34943
I20260929 18:44:40.519979 20283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:44:40.525951 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 20283
I20260929 18:44:40.526105 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:44:40.526443 7797 external_mini_cluster.cc:1428] Setting key f70c13e24e328f3126652ec7c8f83f69
I20260929 18:44:40.529047 20412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:34943 every 8 connection(s)
Sep 29 18:44:40 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:40.544185 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:40.524631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:33527 (local address 127.7.157.126:40455)
0929 18:44:40.524776 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:40.524781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:40.527052 (+ 2271us) server_negotiation.cc:407] Connection header received
0929 18:44:40.528494 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:40.528498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:40.528552 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:40.528605 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:40.530339 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:40.531214 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:40.533001 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:40.533380 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:40.533459 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:40.537192 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:40.537197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:40.537199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:40.537228 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:40.538910 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:40.539587 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:40.539591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:40.539592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:40.539684 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:40.540105 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:40.540108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:40.540110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:40.540353 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:40.540445 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:40.543848 (+ 3403us) server_negotiation.cc:299] Negotiation successful
0929 18:44:40.543996 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:44:40.546173 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:40.523395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:40455 (local address 127.7.157.66:33527)
0929 18:44:40.526825 (+ 3430us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:40.526903 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:40.528228 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:40.529060 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:40.529070 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:40.529436 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:40.530152 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:40.530175 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:40.531372 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:40.531375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:40.532861 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:40.532872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:40.533697 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:40.533701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:40.534058 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:40.535097 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:40.535134 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:40.536968 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:40.539056 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:40.539062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:40.539109 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:40.539467 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:40.539800 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:40.539803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:40.539805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:40.539993 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:40.540513 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:40.540520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:40.540859 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 18:44:40.544676 (+ 3817us) client_negotiation.cc:240] Negotiation successful
0929 18:44:40.545051 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3150,"thread_start_us":156,"threads_started":1}
I20260929 18:44:40.547775 20413 heartbeater.cc:347] Connected to a master server at 127.7.157.126:40455
I20260929 18:44:40.548020 20413 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:40.548643 20413 heartbeater.cc:511] Master 127.7.157.126:40455 requested a full tablet report, sending...
I20260929 18:44:40.549863 20063 ts_manager.cc:194] Registered new tserver with Master: ceb7932d7a32449cb24bfdde7f67d33d (127.7.157.66:34943)
I20260929 18:44:40.550650 20063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:33527
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:44:40.590354 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.157.84:45107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:40455
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:44:40 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
W20260929 18:44:40.779781 20418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:40.780169 20418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:40.780227 20418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:40.786056 20418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:40.786149 20418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:40.786232 20418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:44:40.793910 20418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.157.84:45107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:40455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:40.795284 20418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:40.796581 20418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:40.810920 20424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:40.811357 20418 server_base.cc:1034] running on GCE node
W20260929 18:44:40.811677 20423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:40.811935 20426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:40.812438 20418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:40.813236 20418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:40.814412 20418 hybrid_clock.cc:648] HybridClock initialized: now 1790707480814348 us; error 81 us; skew 500 ppm
Sep 29 18:44:40 dist-test-slave-smpn krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:40.817947 20418 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:44:40.818985 20418 webserver.cc:460] Webserver started at http://127.7.157.67:35803/ using document root <none> and password file <none>
I20260929 18:44:40.819669 20418 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:40.819733 20418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:40.819931 20418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:44:40 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:44:40.844292 20418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "2d9fc563cb454e07bb06a3a7d06c20cc"
format_stamp: "Formatted at 2026-09-29 18:44:40 on dist-test-slave-smpn"
server_key: "ef12b9572ee4ef4e1485139ecc30a3e3"
server_key_iv: "8435ab4668784ec6832a208a1d22a814"
server_key_version: "kuduclusterkey@0"
I20260929 18:44:40.845748 20418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "2d9fc563cb454e07bb06a3a7d06c20cc"
format_stamp: "Formatted at 2026-09-29 18:44:40 on dist-test-slave-smpn"
server_key: "ef12b9572ee4ef4e1485139ecc30a3e3"
server_key_iv: "8435ab4668784ec6832a208a1d22a814"
server_key_version: "kuduclusterkey@0"
I20260929 18:44:40.851393 20418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 18:44:40.869017 20433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:40.870657 20418 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:44:40.870771 20418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "2d9fc563cb454e07bb06a3a7d06c20cc"
format_stamp: "Formatted at 2026-09-29 18:44:40 on dist-test-slave-smpn"
server_key: "ef12b9572ee4ef4e1485139ecc30a3e3"
server_key_iv: "8435ab4668784ec6832a208a1d22a814"
server_key_version: "kuduclusterkey@0"
I20260929 18:44:40.870891 20418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:40.912853 20418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:40.916369 20418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:40.916601 20418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:40.917272 20418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:44:40.918298 20418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:44:40.918403 20418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:40.918494 20418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:44:40.918556 20418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:40.930784 20418 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:37701
I20260929 18:44:40.931965 20418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:44:40.932533 20546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:37701 every 8 connection(s)
I20260929 18:44:40.934685 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 20418
I20260929 18:44:40.934788 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:44:40.935130 7797 external_mini_cluster.cc:1428] Setting key c538937d04cec5643eaf39b4e61a89c9
Sep 29 18:44:40 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707480, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:40.949826 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:40.936470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:42129 (local address 127.7.157.126:40455)
0929 18:44:40.936628 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:40.936633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:40.937089 (+ 456us) server_negotiation.cc:407] Connection header received
0929 18:44:40.938446 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:40.938449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:40.938499 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:40.938546 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:40.940092 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:40.940852 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:40.942463 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:40.942780 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:40.942846 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:40.945850 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:40.945854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:40.945856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:40.945882 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:40.947444 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:40.948051 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:40.948055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:40.948056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:40.948144 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:40.948524 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:40.948528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:40.948529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:40.948763 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:40.948860 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:40.949546 (+ 686us) server_negotiation.cc:299] Negotiation successful
0929 18:44:40.949665 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 18:44:40.951828 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:40.935357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:40455 (local address 127.7.157.67:42129)
0929 18:44:40.936944 (+ 1587us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:40.937013 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:40.938018 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:40.938922 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:40.938929 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:40.939313 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:40.939930 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:40.939948 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:40.941009 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:40.941013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:40.942337 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:40.942347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:40.943041 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:40.943045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:40.943261 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:40.943889 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:40.943909 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:40.945670 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:40.947578 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:40.947584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:40.947601 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:40.947930 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:40.948246 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:40.948249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:40.948251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:40.948427 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:40.948945 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:40.948952 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:40.949281 (+ 329us) client_negotiation.cc:769] Sending connection context
0929 18:44:40.950262 (+ 981us) client_negotiation.cc:240] Negotiation successful
0929 18:44:40.950648 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1382,"thread_start_us":139,"threads_started":1}
I20260929 18:44:40.953258 20547 heartbeater.cc:347] Connected to a master server at 127.7.157.126:40455
I20260929 18:44:40.953505 20547 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:40.954062 20547 heartbeater.cc:511] Master 127.7.157.126:40455 requested a full tablet report, sending...
I20260929 18:44:40.955219 20063 ts_manager.cc:194] Registered new tserver with Master: 2d9fc563cb454e07bb06a3a7d06c20cc (127.7.157.67:37701)
I20260929 18:44:40.955976 20063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:42129
I20260929 18:44:40.961021 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:44:40.961952 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 20144
I20260929 18:44:40.968906 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 20283
I20260929 18:44:40.976394 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 20418
I20260929 18:44:40.984709 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 20031
I20260929 18:44:40.993727 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:40455
--webserver_interface=127.7.157.126
--webserver_port=38407
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:40455
--ranger_config_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.157.84:45107/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.157.84:45107/kms with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:44:41.103117 20550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:41.103374 20550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:41.103412 20550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:41.107326 20550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:44:41.107380 20550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:41.107398 20550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:41.107411 20550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:44:41.107431 20550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:44:41.112787 20550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.157.84:45107/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:40455
--ranger_config_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:40455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=38407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:41.113770 20550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:41.114718 20550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:41.122646 20555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:41.122741 20556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:41.122653 20558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:41.122890 20550 server_base.cc:1034] running on GCE node
I20260929 18:44:41.123471 20550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:41.124495 20550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:41.125690 20550 hybrid_clock.cc:648] HybridClock initialized: now 1790707481125675 us; error 25 us; skew 500 ppm
Sep 29 18:44:41 dist-test-slave-smpn krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707481, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:41.129173 20550 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:44:41.130141 20550 webserver.cc:460] Webserver started at http://127.7.157.126:38407/ using document root <none> and password file <none>
I20260929 18:44:41.130678 20550 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:41.130754 20550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:44:41 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707481, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:44:41.190652 20550 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:44:41.192961 20566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:41.194262 20550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:44:41.194388 20550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "944ac7088b084cf79e140025059c49fa"
format_stamp: "Formatted at 2026-09-29 18:44:39 on dist-test-slave-smpn"
server_key: "0f52ae0e11a1beb674c799ffbb7fe26a"
server_key_iv: "e4fd347324d3ecd7f835926084b930e3"
server_key_version: "kuduclusterkey@0"
I20260929 18:44:41.194962 20550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:41.207278 20550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:41.209692 20550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:41.209862 20550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:41.216879 20550 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:40455
I20260929 18:44:41.216885 20618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:40455 every 8 connection(s)
I20260929 18:44:41.217945 20550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:44:41.220827 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 20550
I20260929 18:44:41.221254 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.157.84:45107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:33095
--local_ip_for_outbound_sockets=127.7.157.65
--tserver_master_addrs=127.7.157.126:40455
--webserver_port=45865
--webserver_interface=127.7.157.65
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.157.84:45107/kms with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:44:41.222246 20619 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:44:41.223937 20619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa: Bootstrap starting.
I20260929 18:44:41.239763 20619 log.cc:824] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:41.245921 20619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:44:41.246258 20619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa: Bootstrap complete.
I20260929 18:44:41.250036 20619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944ac7088b084cf79e140025059c49fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40455 } }
I20260929 18:44:41.250422 20619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944ac7088b084cf79e140025059c49fa, State: Initialized, Role: FOLLOWER
I20260929 18:44:41.250900 20619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944ac7088b084cf79e140025059c49fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40455 } }
I20260929 18:44:41.251024 20619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:44:41.251108 20619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:44:41.251202 20619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:44:41.252967 20619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944ac7088b084cf79e140025059c49fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40455 } }
I20260929 18:44:41.253263 20619 leader_election.cc:302] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 944ac7088b084cf79e140025059c49fa; no voters:
I20260929 18:44:41.253537 20619 leader_election.cc:288] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:44:41.253635 20623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:44:41.253825 20623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [term 2 LEADER]: Becoming Leader. State: Replica: 944ac7088b084cf79e140025059c49fa, State: Running, Role: LEADER
I20260929 18:44:41.254101 20623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944ac7088b084cf79e140025059c49fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40455 } }
I20260929 18:44:41.254684 20619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:44:41.255012 20625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 944ac7088b084cf79e140025059c49fa. Latest consensus state: current_term: 2 leader_uuid: "944ac7088b084cf79e140025059c49fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944ac7088b084cf79e140025059c49fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40455 } } }
I20260929 18:44:41.255167 20625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [sys.catalog]: This master's current role is: LEADER
I20260929 18:44:41.255381 20624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "944ac7088b084cf79e140025059c49fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944ac7088b084cf79e140025059c49fa" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 40455 } } }
I20260929 18:44:41.255493 20624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa [sys.catalog]: This master's current role is: LEADER
I20260929 18:44:41.255774 20627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:44:41.258783 20627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:44:41.259349 20619 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:44:41.259420 20627 catalog_manager.cc:1259] Loaded cluster ID: 070860578bfc494695cfd843fdcdeecc
I20260929 18:44:41.259454 20627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:44:41.261225 20627 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:44:41.261780 20627 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa: Loaded TSK: 0
I20260929 18:44:41.262732 20627 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:44:41.337608 20621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:41.337924 20621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:41.337987 20621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:41.342381 20621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:41.342450 20621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:41.342533 20621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:44:41.348510 20621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.157.84:45107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:33095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=45865
--tserver_master_addrs=127.7.157.126:40455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:41.349855 20621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:41.350979 20621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:41.364133 20656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:41.364064 20659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:41.364202 20657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:41.364661 20621 server_base.cc:1034] running on GCE node
I20260929 18:44:41.365068 20621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:41.365793 20621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:41.366948 20621 hybrid_clock.cc:648] HybridClock initialized: now 1790707481366919 us; error 35 us; skew 500 ppm
Sep 29 18:44:41 dist-test-slave-smpn krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707481, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:41.369678 20621 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:44:41.370556 20621 webserver.cc:460] Webserver started at http://127.7.157.65:45865/ using document root <none> and password file <none>
I20260929 18:44:41.371110 20621 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:41.371203 20621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:44:41 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707481, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:44:41.396427 20621 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:44:41.406579 20667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:41.408357 20621 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:44:41.408565 20621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "79f787b4b1574c6eb68f0183f6112890"
format_stamp: "Formatted at 2026-09-29 18:44:40 on dist-test-slave-smpn"
server_key: "754592c54464246e7646887f97d23481"
server_key_iv: "83c2b264819c1c5a29515169f035aa79"
server_key_version: "kuduclusterkey@0"
I20260929 18:44:41.409385 20621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:41.438905 20621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:41.441941 20621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:41.442130 20621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:41.442817 20621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:44:41.443881 20621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:44:41.443941 20621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:41.443994 20621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:44:41.444015 20621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:41.457731 20621 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:33095
I20260929 18:44:41.459137 20621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:44:41.464468 20780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:33095 every 8 connection(s)
I20260929 18:44:41.468354 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 20621
I20260929 18:44:41.468813 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.157.84:45107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:34943
--local_ip_for_outbound_sockets=127.7.157.66
--tserver_master_addrs=127.7.157.126:40455
--webserver_port=38099
--webserver_interface=127.7.157.66
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.157.84:45107/kms with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:44:41 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707481, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:41.486299 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:41.467048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:40455 (local address 127.7.157.65:40477)
0929 18:44:41.469332 (+ 2284us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:41.469393 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:41.470495 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:41.472162 (+ 1667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:41.472183 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:41.472544 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:41.473228 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:41.473247 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:41.475479 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:41.475482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:41.476885 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:41.476897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:41.477956 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:41.477960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:41.478114 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:41.478730 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:41.478746 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:41.480206 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:41.482619 (+ 2413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:41.482626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:41.482640 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:41.482985 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:41.483452 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:41.483455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:41.483457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:41.483629 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:41.484363 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:41.484369 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:41.484669 (+ 300us) client_negotiation.cc:769] Sending connection context
0929 18:44:41.484858 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 18:44:41.485210 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1406,"thread_start_us":151,"threads_started":1}
I20260929 18:44:41.488864 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:41.467616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:40477 (local address 127.7.157.126:40455)
0929 18:44:41.470741 (+ 3125us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:41.470764 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:41.470819 (+ 55us) server_negotiation.cc:407] Connection header received
0929 18:44:41.471158 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:41.471182 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:41.471631 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:41.473477 (+ 1846us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:41.473802 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:41.475328 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:41.477036 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:41.477771 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:41.478456 (+ 685us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:41.480430 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:41.480437 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:41.480453 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:41.480485 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:41.482485 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:41.483165 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:41.483183 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:41.483189 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:41.483335 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:41.483743 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:41.483746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:41.483748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:41.484230 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:41.487441 (+ 3211us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:41.487606 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 18:44:41.487914 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2953,"thread_start_us":128,"threads_started":1}
I20260929 18:44:41.491899 20781 heartbeater.cc:347] Connected to a master server at 127.7.157.126:40455
I20260929 18:44:41.492161 20781 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:41.492761 20781 heartbeater.cc:511] Master 127.7.157.126:40455 requested a full tablet report, sending...
W20260929 18:44:41.495141 20781 heartbeater.cc:644] Failed to heartbeat to 127.7.157.126:40455 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:44:41.495275 20781 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:41.495452 20781 heartbeater.cc:511] Master 127.7.157.126:40455 requested a full tablet report, sending...
I20260929 18:44:41.496484 20781 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:41.496641 20781 heartbeater.cc:511] Master 127.7.157.126:40455 requested a full tablet report, sending...
W20260929 18:44:41.497475 20781 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:44:41.609045 20785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:41.609297 20785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:41.609338 20785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:41.613777 20785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:41.613859 20785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:41.613945 20785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:44:41.621311 20785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.157.84:45107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:34943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=38099
--tserver_master_addrs=127.7.157.126:40455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:41.622776 20785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:41.624154 20785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:41.634202 20791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:41.634207 20793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:41.634459 20785 server_base.cc:1034] running on GCE node
W20260929 18:44:41.634213 20790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:41.635015 20785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:41.635705 20785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:41.636889 20785 hybrid_clock.cc:648] HybridClock initialized: now 1790707481636862 us; error 43 us; skew 500 ppm
Sep 29 18:44:41 dist-test-slave-smpn krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707481, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:41.640259 20785 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:44:41.641369 20785 webserver.cc:460] Webserver started at http://127.7.157.66:38099/ using document root <none> and password file <none>
I20260929 18:44:41.641949 20785 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:41.642033 20785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:44:41 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707481, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:44:41.671499 20785 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:44:41.674631 20801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:41.676363 20785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:44:41.676481 20785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "ceb7932d7a32449cb24bfdde7f67d33d"
format_stamp: "Formatted at 2026-09-29 18:44:40 on dist-test-slave-smpn"
server_key: "dd2639c86418a51b0c4f04ede2d21543"
server_key_iv: "ca64d4f11a30e23e748dcb2090397e18"
server_key_version: "kuduclusterkey@0"
I20260929 18:44:41.677210 20785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:41.702335 20785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:41.705758 20785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:41.705945 20785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:41.706578 20785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:44:41.707499 20785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:44:41.707549 20785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:41.707602 20785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:44:41.707630 20785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:41.717509 20785 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:34943
I20260929 18:44:41.717552 20914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:34943 every 8 connection(s)
I20260929 18:44:41.718544 20785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:44:41.720696 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 20785
I20260929 18:44:41.721127 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.157.84:45107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:37701
--local_ip_for_outbound_sockets=127.7.157.67
--tserver_master_addrs=127.7.157.126:40455
--webserver_port=35803
--webserver_interface=127.7.157.67
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.157.84:45107/kms with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:44:41 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707481, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:41.734513 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:41.722352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:52205 (local address 127.7.157.126:40455)
0929 18:44:41.722604 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:41.722608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:41.723005 (+ 397us) server_negotiation.cc:407] Connection header received
0929 18:44:41.724160 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:41.724165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:41.724221 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:41.724274 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:41.725476 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:41.725989 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:41.727237 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:41.727466 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:41.727552 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:41.730149 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:41.730153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:41.730156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:41.730180 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:41.732367 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:41.733002 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:41.733006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:41.733008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:41.733090 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:41.733425 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:41.733429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:41.733430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:41.733621 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:41.733707 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:41.734218 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 18:44:41.734348 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260929 18:44:41.735749 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:41.721167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:40455 (local address 127.7.157.66:52205)
0929 18:44:41.722853 (+ 1686us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:41.722893 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:41.723907 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:41.724627 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:41.724634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:41.724880 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:41.725337 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:41.725349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:41.726111 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:41.726114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:41.727117 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:41.727128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:41.727596 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:41.727599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:41.727768 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:41.728377 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:41.728397 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:41.729992 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:41.732513 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:41.732520 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:41.732537 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:41.732875 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:41.733197 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:41.733200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:41.733202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:41.733337 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:41.733720 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:41.733727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:41.733986 (+ 259us) client_negotiation.cc:769] Sending connection context
0929 18:44:41.734165 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 18:44:41.734478 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1502,"thread_start_us":112,"threads_started":1}
I20260929 18:44:41.737254 20915 heartbeater.cc:347] Connected to a master server at 127.7.157.126:40455
I20260929 18:44:41.737490 20915 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:41.738046 20915 heartbeater.cc:511] Master 127.7.157.126:40455 requested a full tablet report, sending...
W20260929 18:44:41.739746 20915 heartbeater.cc:644] Failed to heartbeat to 127.7.157.126:40455 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:44:41.739861 20915 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:41.740010 20915 heartbeater.cc:511] Master 127.7.157.126:40455 requested a full tablet report, sending...
I20260929 18:44:41.740837 20915 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:41.740988 20915 heartbeater.cc:511] Master 127.7.157.126:40455 requested a full tablet report, sending...
W20260929 18:44:41.741714 20915 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:44:41.859455 20918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:41.859704 20918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:41.859766 20918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:41.864068 20918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:41.864149 20918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:41.864235 20918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:44:41.869668 20918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.157.84:45107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:37701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=35803
--tserver_master_addrs=127.7.157.126:40455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:41.870805 20918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:41.872005 20918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:41.883783 20925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:41.883862 20927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:41.884310 20924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:41.884965 20918 server_base.cc:1034] running on GCE node
I20260929 18:44:41.885519 20918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:41.886350 20918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:41.887535 20918 hybrid_clock.cc:648] HybridClock initialized: now 1790707481887500 us; error 50 us; skew 500 ppm
Sep 29 18:44:41 dist-test-slave-smpn krb5kdc[19587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707481, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:41.890254 20918 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:44:41.891162 20918 webserver.cc:460] Webserver started at http://127.7.157.67:35803/ using document root <none> and password file <none>
I20260929 18:44:41.891754 20918 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:41.891839 20918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:44:41 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707481, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:44:41.913883 20918 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.000s
I20260929 18:44:41.917232 20935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:41.918915 20918 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 18:44:41.919049 20918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "2d9fc563cb454e07bb06a3a7d06c20cc"
format_stamp: "Formatted at 2026-09-29 18:44:40 on dist-test-slave-smpn"
server_key: "ef12b9572ee4ef4e1485139ecc30a3e3"
server_key_iv: "8435ab4668784ec6832a208a1d22a814"
server_key_version: "kuduclusterkey@0"
I20260929 18:44:41.919903 20918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:41.933310 20918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:41.936741 20918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:41.936945 20918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:41.937654 20918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:44:41.938699 20918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:44:41.938759 20918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:41.938814 20918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:44:41.938845 20918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:41.950177 20918 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:37701
I20260929 18:44:41.950219 21048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:37701 every 8 connection(s)
I20260929 18:44:41.951491 20918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:44:41.957762 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 20918
Sep 29 18:44:41 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707481, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:41.967913 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:41.953324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:46179 (local address 127.7.157.126:40455)
0929 18:44:41.953442 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:41.953445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:41.954508 (+ 1063us) server_negotiation.cc:407] Connection header received
0929 18:44:41.955530 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:41.955533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:41.955591 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:41.955658 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:41.956969 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:41.957775 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:41.959488 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:41.959835 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:41.959919 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:41.963673 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:41.963677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:41.963680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:41.963708 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:41.965355 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:41.965866 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:41.965870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:41.965872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:41.965964 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:41.966381 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:41.966385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:41.966387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:41.966607 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:41.966687 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:41.967608 (+ 921us) server_negotiation.cc:299] Negotiation successful
0929 18:44:41.967748 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:44:41.968700 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:41.953860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:40455 (local address 127.7.157.67:46179)
0929 18:44:41.954363 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:41.954404 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:41.955315 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:41.955829 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:41.955835 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:41.956094 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:41.956788 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:41.956808 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:41.957921 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:41.957925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:41.959333 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:41.959344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:41.960286 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:41.960290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:41.960500 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:41.961120 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:41.961141 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:41.963469 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:41.965458 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:41.965462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:41.965485 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:41.965770 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:41.966073 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:41.966076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:41.966079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:41.966270 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:41.966755 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:41.966761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:41.967108 (+ 347us) client_negotiation.cc:769] Sending connection context
0929 18:44:41.967324 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 18:44:41.967670 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":137,"threads_started":1}
I20260929 18:44:41.970002 21049 heartbeater.cc:347] Connected to a master server at 127.7.157.126:40455
I20260929 18:44:41.970252 21049 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:41.970731 21049 heartbeater.cc:511] Master 127.7.157.126:40455 requested a full tablet report, sending...
I20260929 18:44:41.971365 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:41.958798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.7.157.126:40455)
0929 18:44:41.959744 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:41.959754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:41.959765 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:44:41.959797 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:41.959800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:41.959842 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:41.959916 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:41.961184 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:41.962037 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:41.964487 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:41.964806 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:41.964881 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:41.966719 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:41.966722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:41.966724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:41.966744 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:41.968192 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:41.968613 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:41.968616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:41.968617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:41.968689 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:41.969053 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:41.969056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:41.969058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:41.969248 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:41.969421 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:41.969790 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 18:44:41.969899 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":74,"threads_started":1}
W20260929 18:44:41.972358 21049 heartbeater.cc:644] Failed to heartbeat to 127.7.157.126:40455 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:44:41.972466 21049 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:41.972630 21049 heartbeater.cc:511] Master 127.7.157.126:40455 requested a full tablet report, sending...
I20260929 18:44:41.973421 21049 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:41.973582 21049 heartbeater.cc:511] Master 127.7.157.126:40455 requested a full tablet report, sending...
W20260929 18:44:41.974287 21049 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 18:44:42 dist-test-slave-smpn krb5kdc[19587](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790707482, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:42.497732 20781 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:42.498020 20781 heartbeater.cc:511] Master 127.7.157.126:40455 requested a full tablet report, sending...
I20260929 18:44:42.499228 20582 ts_manager.cc:194] Registered new tserver with Master: 79f787b4b1574c6eb68f0183f6112890 (127.7.157.65:33095)
I20260929 18:44:42.500489 20582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:40477
I20260929 18:44:42.741932 20915 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:42.742287 20915 heartbeater.cc:511] Master 127.7.157.126:40455 requested a full tablet report, sending...
I20260929 18:44:42.743235 20582 ts_manager.cc:194] Registered new tserver with Master: ceb7932d7a32449cb24bfdde7f67d33d (127.7.157.66:34943)
I20260929 18:44:42.743821 20582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:52205
Sep 29 18:44:42 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707482, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:44:42.974516 21049 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:42.974838 21049 heartbeater.cc:511] Master 127.7.157.126:40455 requested a full tablet report, sending...
I20260929 18:44:42.975831 20583 ts_manager.cc:194] Registered new tserver with Master: 2d9fc563cb454e07bb06a3a7d06c20cc (127.7.157.67:37701)
I20260929 18:44:42.976590 20583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:46179
I20260929 18:44:42.976643 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:44:42.992558 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:42.978338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.7.157.126:40455)
0929 18:44:42.978917 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:42.978922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:42.979291 (+ 369us) server_negotiation.cc:407] Connection header received
0929 18:44:42.979334 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:42.979337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:42.979385 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:42.979460 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:42.980381 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:42.981175 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:42.982304 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:42.982654 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:42.982745 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:42.988225 (+ 5480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:42.988230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:42.988232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:42.988265 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:42.990097 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:42.990635 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:42.990638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:42.990640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:42.990740 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:42.991091 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:42.991095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:42.991096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:42.991335 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:42.991462 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:42.992229 (+ 767us) server_negotiation.cc:299] Negotiation successful
0929 18:44:42.992364 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":93,"threads_started":1}
I20260929 18:44:42.996830 20583 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:44:43.001353 20583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 29 18:44:43 dist-test-slave-smpn krb5kdc[19587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707482, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for HTTP/127.7.157.84@KRBTEST.COM
I20260929 18:44:43.029060 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:43.013924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:33095 (local address 127.0.0.1:44808)
0929 18:44:43.015556 (+ 1632us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:43.015591 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:43.015715 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:43.018642 (+ 2927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:43.018646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:43.018673 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:44:43.018789 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:43.018795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.021512 (+ 2717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.021516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:43.024442 (+ 2926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:43.024453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.028414 (+ 3961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.028417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:43.028552 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:43.028603 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 18:44:43.028734 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 18:44:43.028887 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1589,"spinlock_wait_cycles":6528,"thread_start_us":105,"threads_started":1}
I20260929 18:44:43.029714 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:43.013617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44808 (local address 127.7.157.65:33095)
0929 18:44:43.016139 (+ 2522us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:43.016146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:43.016169 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:44:43.016234 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:43.016238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:43.016397 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:43.016529 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:44:43.020157 (+ 3628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.021375 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.024591 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.025466 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.025573 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:43.025754 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:43.029419 (+ 3665us) server_negotiation.cc:299] Negotiation successful
0929 18:44:43.029548 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2427,"thread_start_us":98,"threads_started":1}
I20260929 18:44:43.032717 21082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:43.014985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.7.157.66:34943)
0929 18:44:43.015967 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:43.015974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:43.016863 (+ 889us) server_negotiation.cc:407] Connection header received
0929 18:44:43.018105 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:43.018111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:43.018304 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:43.018405 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:44:43.018799 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.020026 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.030415 (+ 10389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.031317 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.031404 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:43.031562 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:43.032207 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 18:44:43.032519 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886,"thread_start_us":97,"threads_started":1}
I20260929 18:44:43.032099 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:43.014586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:34943 (local address 127.0.0.1:53372)
0929 18:44:43.016772 (+ 2186us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:43.016785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:43.017643 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:43.018564 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:43.018567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:43.018583 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:44:43.018679 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:43.018685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.022239 (+ 3554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.022243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:43.023437 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:43.023450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.031686 (+ 8236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.031690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:43.031791 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:43.031818 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 18:44:43.031887 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:44:43.031963 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2126,"thread_start_us":94,"threads_started":1}
I20260929 18:44:43.036393 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:43.013910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54076 (local address 127.7.157.67:37701)
0929 18:44:43.017052 (+ 3142us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:43.017058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:43.017080 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:44:43.017154 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:43.017160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:43.017356 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:43.017491 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:44:43.025841 (+ 8350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.026950 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.033147 (+ 6197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.034018 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.034127 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:43.034285 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:43.036140 (+ 1855us) server_negotiation.cc:299] Negotiation successful
0929 18:44:43.036260 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3042,"thread_start_us":76,"threads_started":1}
I20260929 18:44:43.038496 20715 tablet_service.cc:1467] Processing CreateTablet for tablet 192bcc1b0a054651bd5e178b9087873f (DEFAULT_TABLE table=test-table [id=d9fe700df0e14d7b9884e0064b788b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:44:43.039814 20715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192bcc1b0a054651bd5e178b9087873f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:44:43.043588 20834 tablet_service.cc:1467] Processing CreateTablet for tablet 192bcc1b0a054651bd5e178b9087873f (DEFAULT_TABLE table=test-table [id=d9fe700df0e14d7b9884e0064b788b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:44:43.044744 20834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192bcc1b0a054651bd5e178b9087873f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:44:43.034782 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:43.013119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:37701 (local address 127.0.0.1:54076)
0929 18:44:43.015538 (+ 2419us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:43.015591 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:43.015716 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:43.017769 (+ 2053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:43.017775 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:43.017797 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:44:43.017932 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:43.017938 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.027108 (+ 9170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.027111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:43.028270 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:43.028282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.034359 (+ 6077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.034363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:43.034456 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:43.034481 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:44:43.034552 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 18:44:43.034630 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2098,"thread_start_us":107,"threads_started":1}
I20260929 18:44:43.050985 20983 tablet_service.cc:1467] Processing CreateTablet for tablet 192bcc1b0a054651bd5e178b9087873f (DEFAULT_TABLE table=test-table [id=d9fe700df0e14d7b9884e0064b788b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:44:43.052038 20983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192bcc1b0a054651bd5e178b9087873f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:44:43.059785 21083 tablet_bootstrap.cc:492] T 192bcc1b0a054651bd5e178b9087873f P 79f787b4b1574c6eb68f0183f6112890: Bootstrap starting.
I20260929 18:44:43.061681 21085 tablet_bootstrap.cc:492] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc: Bootstrap starting.
I20260929 18:44:43.064218 21085 tablet_bootstrap.cc:654] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc: Neither blocks nor log segments found. Creating new log.
I20260929 18:44:43.065240 21085 log.cc:824] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:43.065726 21084 tablet_bootstrap.cc:492] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d: Bootstrap starting.
I20260929 18:44:43.068151 21083 tablet_bootstrap.cc:654] T 192bcc1b0a054651bd5e178b9087873f P 79f787b4b1574c6eb68f0183f6112890: Neither blocks nor log segments found. Creating new log.
I20260929 18:44:43.068145 21084 tablet_bootstrap.cc:654] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d: Neither blocks nor log segments found. Creating new log.
I20260929 18:44:43.069053 21083 log.cc:824] T 192bcc1b0a054651bd5e178b9087873f P 79f787b4b1574c6eb68f0183f6112890: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:43.071692 21085 tablet_bootstrap.cc:492] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc: No bootstrap required, opened a new log
I20260929 18:44:43.071849 21083 tablet_bootstrap.cc:492] T 192bcc1b0a054651bd5e178b9087873f P 79f787b4b1574c6eb68f0183f6112890: No bootstrap required, opened a new log
I20260929 18:44:43.071926 21085 ts_tablet_manager.cc:1397] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260929 18:44:43.072038 21083 ts_tablet_manager.cc:1397] T 192bcc1b0a054651bd5e178b9087873f P 79f787b4b1574c6eb68f0183f6112890: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260929 18:44:43.076380 21083 raft_consensus.cc:348] T 192bcc1b0a054651bd5e178b9087873f P 79f787b4b1574c6eb68f0183f6112890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9fc563cb454e07bb06a3a7d06c20cc" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37701 } } peers { permanent_uuid: "79f787b4b1574c6eb68f0183f6112890" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 33095 } } peers { permanent_uuid: "ceb7932d7a32449cb24bfdde7f67d33d" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 34943 } }
I20260929 18:44:43.076690 21083 raft_consensus.cc:374] T 192bcc1b0a054651bd5e178b9087873f P 79f787b4b1574c6eb68f0183f6112890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:44:43.076743 21083 raft_consensus.cc:729] T 192bcc1b0a054651bd5e178b9087873f P 79f787b4b1574c6eb68f0183f6112890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f787b4b1574c6eb68f0183f6112890, State: Initialized, Role: FOLLOWER
I20260929 18:44:43.076920 21084 log.cc:824] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:43.077564 21083 consensus_queue.cc:260] T 192bcc1b0a054651bd5e178b9087873f P 79f787b4b1574c6eb68f0183f6112890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9fc563cb454e07bb06a3a7d06c20cc" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37701 } } peers { permanent_uuid: "79f787b4b1574c6eb68f0183f6112890" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 33095 } } peers { permanent_uuid: "ceb7932d7a32449cb24bfdde7f67d33d" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 34943 } }
I20260929 18:44:43.077751 21085 raft_consensus.cc:348] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9fc563cb454e07bb06a3a7d06c20cc" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37701 } } peers { permanent_uuid: "79f787b4b1574c6eb68f0183f6112890" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 33095 } } peers { permanent_uuid: "ceb7932d7a32449cb24bfdde7f67d33d" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 34943 } }
I20260929 18:44:43.078048 21085 raft_consensus.cc:374] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:44:43.078110 21085 raft_consensus.cc:729] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d9fc563cb454e07bb06a3a7d06c20cc, State: Initialized, Role: FOLLOWER
I20260929 18:44:43.078603 21085 consensus_queue.cc:260] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9fc563cb454e07bb06a3a7d06c20cc" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37701 } } peers { permanent_uuid: "79f787b4b1574c6eb68f0183f6112890" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 33095 } } peers { permanent_uuid: "ceb7932d7a32449cb24bfdde7f67d33d" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 34943 } }
I20260929 18:44:43.079015 21083 ts_tablet_manager.cc:1428] T 192bcc1b0a054651bd5e178b9087873f P 79f787b4b1574c6eb68f0183f6112890: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:44:43.079474 20781 heartbeater.cc:503] Master 127.7.157.126:40455 was elected leader, sending a full tablet report...
I20260929 18:44:43.080758 21085 ts_tablet_manager.cc:1428] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260929 18:44:43.084486 21084 tablet_bootstrap.cc:492] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d: No bootstrap required, opened a new log
I20260929 18:44:43.084719 21084 ts_tablet_manager.cc:1397] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20260929 18:44:43.089102 21084 raft_consensus.cc:348] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9fc563cb454e07bb06a3a7d06c20cc" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37701 } } peers { permanent_uuid: "79f787b4b1574c6eb68f0183f6112890" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 33095 } } peers { permanent_uuid: "ceb7932d7a32449cb24bfdde7f67d33d" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 34943 } }
I20260929 18:44:43.089425 21084 raft_consensus.cc:374] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:44:43.089491 21084 raft_consensus.cc:729] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb7932d7a32449cb24bfdde7f67d33d, State: Initialized, Role: FOLLOWER
I20260929 18:44:43.090481 21049 heartbeater.cc:503] Master 127.7.157.126:40455 was elected leader, sending a full tablet report...
I20260929 18:44:43.092594 21084 consensus_queue.cc:260] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9fc563cb454e07bb06a3a7d06c20cc" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37701 } } peers { permanent_uuid: "79f787b4b1574c6eb68f0183f6112890" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 33095 } } peers { permanent_uuid: "ceb7932d7a32449cb24bfdde7f67d33d" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 34943 } }
I20260929 18:44:43.093752 21084 ts_tablet_manager.cc:1428] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260929 18:44:43.094086 20915 heartbeater.cc:503] Master 127.7.157.126:40455 was elected leader, sending a full tablet report...
I20260929 18:44:43.105291 21092 raft_consensus.cc:482] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:44:43.105753 21092 raft_consensus.cc:504] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9fc563cb454e07bb06a3a7d06c20cc" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37701 } } peers { permanent_uuid: "79f787b4b1574c6eb68f0183f6112890" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 33095 } } peers { permanent_uuid: "ceb7932d7a32449cb24bfdde7f67d33d" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 34943 } }
I20260929 18:44:43.109690 21092 leader_election.cc:288] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d9fc563cb454e07bb06a3a7d06c20cc (127.7.157.67:37701), 79f787b4b1574c6eb68f0183f6112890 (127.7.157.65:33095)
I20260929 18:44:43.114759 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:43.108881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:48955 (local address 127.7.157.65:33095)
0929 18:44:43.109014 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:43.109019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:43.110451 (+ 1432us) server_negotiation.cc:407] Connection header received
0929 18:44:43.110629 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:43.110632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:43.110676 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:43.110718 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:44:43.111117 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.111867 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.113346 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.113952 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.114066 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:43.114137 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:43.114525 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 18:44:43.114623 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:44:43.115479 21094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:43.108335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:33095 (local address 127.7.157.66:48955)
0929 18:44:43.110377 (+ 2042us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:43.110389 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:43.110511 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:43.110848 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:43.110851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:43.110866 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:44:43.110973 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:43.110978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.112006 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.112009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:43.113206 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:43.113217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.114247 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.114251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:43.114390 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:43.114415 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:44:43.115193 (+ 778us) client_negotiation.cc:240] Negotiation successful
0929 18:44:43.115275 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1982,"thread_start_us":77,"threads_started":1}
I20260929 18:44:43.116267 20735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192bcc1b0a054651bd5e178b9087873f" candidate_uuid: "ceb7932d7a32449cb24bfdde7f67d33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f787b4b1574c6eb68f0183f6112890" is_pre_election: true
I20260929 18:44:43.116627 20735 raft_consensus.cc:2454] T 192bcc1b0a054651bd5e178b9087873f P 79f787b4b1574c6eb68f0183f6112890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb7932d7a32449cb24bfdde7f67d33d in term 0.
I20260929 18:44:43.117369 20802 leader_election.cc:302] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79f787b4b1574c6eb68f0183f6112890, ceb7932d7a32449cb24bfdde7f67d33d; no voters:
I20260929 18:44:43.117796 21092 raft_consensus.cc:2781] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:44:43.117884 21092 raft_consensus.cc:482] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:44:43.117923 21092 raft_consensus.cc:3037] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:44:43.119277 21092 raft_consensus.cc:504] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9fc563cb454e07bb06a3a7d06c20cc" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37701 } } peers { permanent_uuid: "79f787b4b1574c6eb68f0183f6112890" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 33095 } } peers { permanent_uuid: "ceb7932d7a32449cb24bfdde7f67d33d" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 34943 } }
I20260929 18:44:43.120231 20735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192bcc1b0a054651bd5e178b9087873f" candidate_uuid: "ceb7932d7a32449cb24bfdde7f67d33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f787b4b1574c6eb68f0183f6112890"
I20260929 18:44:43.120358 20735 raft_consensus.cc:3037] T 192bcc1b0a054651bd5e178b9087873f P 79f787b4b1574c6eb68f0183f6112890 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:44:43.121608 20735 raft_consensus.cc:2454] T 192bcc1b0a054651bd5e178b9087873f P 79f787b4b1574c6eb68f0183f6112890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb7932d7a32449cb24bfdde7f67d33d in term 1.
I20260929 18:44:43.122174 20802 leader_election.cc:302] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79f787b4b1574c6eb68f0183f6112890, ceb7932d7a32449cb24bfdde7f67d33d; no voters:
I20260929 18:44:43.122473 21092 leader_election.cc:288] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [CANDIDATE]: Term 1 election: Requested vote from peers 2d9fc563cb454e07bb06a3a7d06c20cc (127.7.157.67:37701), 79f787b4b1574c6eb68f0183f6112890 (127.7.157.65:33095)
I20260929 18:44:43.122617 21092 raft_consensus.cc:2781] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:44:43.122725 21092 raft_consensus.cc:686] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [term 1 LEADER]: Becoming Leader. State: Replica: ceb7932d7a32449cb24bfdde7f67d33d, State: Running, Role: LEADER
I20260929 18:44:43.123147 21092 consensus_queue.cc:237] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9fc563cb454e07bb06a3a7d06c20cc" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37701 } } peers { permanent_uuid: "79f787b4b1574c6eb68f0183f6112890" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 33095 } } peers { permanent_uuid: "ceb7932d7a32449cb24bfdde7f67d33d" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 34943 } }
I20260929 18:44:43.124748 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:43.107881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:52003 (local address 127.7.157.67:37701)
0929 18:44:43.108016 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:43.108020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:43.108183 (+ 163us) server_negotiation.cc:407] Connection header received
0929 18:44:43.109376 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:43.109380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:43.109434 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:43.109484 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:44:43.110351 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.111090 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.123328 (+ 12238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.123984 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.124096 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:43.124152 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:43.124527 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 18:44:43.124615 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:44:43.125239 21093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:43.107379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:37701 (local address 127.7.157.66:52003)
0929 18:44:43.108087 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:43.108102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:43.108859 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:43.109989 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:43.109992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:43.110011 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:44:43.110146 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:43.110152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.111617 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.111620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:43.123135 (+ 11515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:43.123151 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.124269 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.124273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:43.124382 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:43.124406 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:44:43.124984 (+ 578us) client_negotiation.cc:240] Negotiation successful
0929 18:44:43.125076 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":105,"threads_started":1}
I20260929 18:44:43.126125 20986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192bcc1b0a054651bd5e178b9087873f" candidate_uuid: "ceb7932d7a32449cb24bfdde7f67d33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d9fc563cb454e07bb06a3a7d06c20cc" is_pre_election: true
I20260929 18:44:43.126425 20986 raft_consensus.cc:2454] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb7932d7a32449cb24bfdde7f67d33d in term 0.
I20260929 18:44:43.126763 20985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192bcc1b0a054651bd5e178b9087873f" candidate_uuid: "ceb7932d7a32449cb24bfdde7f67d33d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d9fc563cb454e07bb06a3a7d06c20cc"
I20260929 18:44:43.126870 20985 raft_consensus.cc:3037] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:44:43.128232 20985 raft_consensus.cc:2454] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb7932d7a32449cb24bfdde7f67d33d in term 1.
I20260929 18:44:43.131461 20583 catalog_manager.cc:5495] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb7932d7a32449cb24bfdde7f67d33d (127.7.157.66). New cstate: current_term: 1 leader_uuid: "ceb7932d7a32449cb24bfdde7f67d33d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d9fc563cb454e07bb06a3a7d06c20cc" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79f787b4b1574c6eb68f0183f6112890" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 33095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb7932d7a32449cb24bfdde7f67d33d" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 34943 } health_report { overall_health: HEALTHY } } }
I20260929 18:44:43.166960 21082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:43.161699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.7.157.66:34943)
0929 18:44:43.161816 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:43.161821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:43.161840 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:44:43.162049 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:43.162052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:43.162129 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:43.162252 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:44:43.162670 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.163507 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.164758 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:43.165048 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:43.165138 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:43.165576 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:44:43.166179 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:44:43.166298 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:43.166728 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 18:44:43.166793 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:44:43.174312 20735 raft_consensus.cc:1261] T 192bcc1b0a054651bd5e178b9087873f P 79f787b4b1574c6eb68f0183f6112890 [term 1 FOLLOWER]: Refusing update from remote peer ceb7932d7a32449cb24bfdde7f67d33d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:44:43.174311 20985 raft_consensus.cc:1261] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc [term 1 FOLLOWER]: Refusing update from remote peer ceb7932d7a32449cb24bfdde7f67d33d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:44:43.175160 21095 consensus_queue.cc:1035] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79f787b4b1574c6eb68f0183f6112890" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 33095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:44:43.175992 21095 consensus_queue.cc:1035] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d9fc563cb454e07bb06a3a7d06c20cc" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:44:43.191881 21098 mvcc.cc:205] Tried to move back new op lower bound from 7334737851072270336 to 7334737850876092416. Current Snapshot: MvccSnapshot[applied={T|T < 7334737851072270336}]
I20260929 18:44:43.193320 21099 mvcc.cc:205] Tried to move back new op lower bound from 7334737851072270336 to 7334737850876092416. Current Snapshot: MvccSnapshot[applied={T|T < 7334737851072270336}]
I20260929 18:44:43.196075 21100 mvcc.cc:205] Tried to move back new op lower bound from 7334737851072270336 to 7334737850876092416. Current Snapshot: MvccSnapshot[applied={T|T < 7334737851072270336}]
I20260929 18:44:43.204665 20582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:44:43.204860 20582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:44:43.207650 20582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 944ac7088b084cf79e140025059c49fa: Sending DeleteTablet for 3 replicas of tablet 192bcc1b0a054651bd5e178b9087873f
I20260929 18:44:43.208551 20834 tablet_service.cc:1514] Processing DeleteTablet for tablet 192bcc1b0a054651bd5e178b9087873f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:44:43 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:53372
I20260929 18:44:43.208813 20983 tablet_service.cc:1514] Processing DeleteTablet for tablet 192bcc1b0a054651bd5e178b9087873f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:44:43 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:54076
I20260929 18:44:43.209319 20715 tablet_service.cc:1514] Processing DeleteTablet for tablet 192bcc1b0a054651bd5e178b9087873f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:44:43 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:44808
I20260929 18:44:43.210731 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 20621
I20260929 18:44:43.222167 21108 tablet_replica.cc:330] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc: stopping tablet replica
I20260929 18:44:43.222580 21108 raft_consensus.cc:2229] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:44:43.222976 21108 raft_consensus.cc:2258] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:44:43.227712 21108 ts_tablet_manager.cc:1905] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:44:43.230687 21108 ts_tablet_manager.cc:1918] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:44:43.230803 21108 log.cc:1196] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/192bcc1b0a054651bd5e178b9087873f
W20260929 18:44:43.234912 20567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:33095 (error 108)
W20260929 18:44:43.242717 21080 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:44:43.235815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:33095 (local address 127.0.0.1:44812)
0929 18:44:43.235960 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:43.235977 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:43.236079 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:43.242578 (+ 6499us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:33095: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 18:44:43.242920 20567 catalog_manager.cc:4584] TS 79f787b4b1574c6eb68f0183f6112890 (127.7.157.65:33095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 192bcc1b0a054651bd5e178b9087873f: Network error: Client connection negotiation failed: client connection to 127.7.157.65:33095: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:44:43.243337 21107 tablet_replica.cc:330] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d: stopping tablet replica
I20260929 18:44:43.243732 21107 raft_consensus.cc:2229] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:44:43.244222 21107 raft_consensus.cc:2258] T 192bcc1b0a054651bd5e178b9087873f P ceb7932d7a32449cb24bfdde7f67d33d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:44:43.244577 21108 ts_tablet_manager.cc:1939] T 192bcc1b0a054651bd5e178b9087873f P 2d9fc563cb454e07bb06a3a7d06c20cc: Deleting consensus metadata
I20260929 18:44:43.246065 20569 catalog_manager.cc:4841] TS 2d9fc563cb454e07bb06a3a7d06c20cc (127.7.157.67:37701): tablet 192bcc1b0a054651bd5e178b9087873f (table test-table [id=d9fe700df0e14d7b9884e0064b788b13]) successfully deleted
I20260929 18:44:43.246394 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 20785
I20260929 18:44:43.253296 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 20918
I20260929 18:44:43.256587 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:43.253542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:34943 (local address 127.0.0.1:53382)
0929 18:44:43.256384 (+ 2842us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:43.256487 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.66:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2747}
I20260929 18:44:43.261720 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 20550
I20260929 18:44:43.267376 7797 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 18:44:43.601425 7797 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 18:44:43.601540 7797 mini_ranger.cc:66] Stopping Ranger...
I20260929 18:44:43.935245 7797 mini_ranger.cc:68] Stopped Ranger
2026-09-29 18:44:43.935 UTC [19617] LOG: received smart shutdown request
2026-09-29 18:44:43.939 UTC [19617] LOG: background worker "logical replication launcher" (PID 19626) exited with exit code 1
2026-09-29 18:44:43.939 UTC [19621] LOG: shutting down
2026-09-29 18:44:44.087 UTC [19617] LOG: database system is shut down
2026-09-29T18:44:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47354 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:44:44 dist-test-slave-smpn krb5kdc[21119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:44:44 dist-test-slave-smpn krb5kdc[21119](info): set up 2 sockets
Sep 29 18:44:44 dist-test-slave-smpn krb5kdc[21119](info): commencing operation
krb5kdc: starting...
W20260929 18:44:45.146862 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:44:45 dist-test-slave-smpn krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:44:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:44:45.259538 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:39303
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:39303,127.7.157.125:42965,127.7.157.124:41413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:44:45.366423 21132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:45.366670 21132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:45.366709 21132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:45.370656 21132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:44:45.370716 21132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:45.370734 21132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:45.370749 21132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:44:45.370762 21132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:44:45.375886 21132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:39303,127.7.157.125:42965,127.7.157.124:41413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:39303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:45.376891 21132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:45.377878 21132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:45.386011 21138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:45.386011 21137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:45.386054 21132 server_base.cc:1034] running on GCE node
W20260929 18:44:45.386008 21140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:45.386845 21132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:45.387979 21132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:45.389165 21132 hybrid_clock.cc:648] HybridClock initialized: now 1790707485389153 us; error 30 us; skew 500 ppm
Sep 29 18:44:45 dist-test-slave-smpn krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:45.395392 21132 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:44:45.396288 21132 webserver.cc:460] Webserver started at http://127.7.157.126:33987/ using document root <none> and password file <none>
I20260929 18:44:45.396824 21132 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:45.396888 21132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:45.397078 21132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:44:45.399139 21132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "72d29c8d68f147d593061cb3a45a637f"
format_stamp: "Formatted at 2026-09-29 18:44:45 on dist-test-slave-smpn"
server_key: "a09b37539dbdf5002a2a89b81cb271f6"
server_key_iv: "61126d3bb081ce229804ea5003f9268e"
server_key_version: "encryptionkey@0"
I20260929 18:44:45.399680 21132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "72d29c8d68f147d593061cb3a45a637f"
format_stamp: "Formatted at 2026-09-29 18:44:45 on dist-test-slave-smpn"
server_key: "a09b37539dbdf5002a2a89b81cb271f6"
server_key_iv: "61126d3bb081ce229804ea5003f9268e"
server_key_version: "encryptionkey@0"
I20260929 18:44:45.403376 21132 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 18:44:45.405783 21147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:45.407063 21132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:44:45.407207 21132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "72d29c8d68f147d593061cb3a45a637f"
format_stamp: "Formatted at 2026-09-29 18:44:45 on dist-test-slave-smpn"
server_key: "a09b37539dbdf5002a2a89b81cb271f6"
server_key_iv: "61126d3bb081ce229804ea5003f9268e"
server_key_version: "encryptionkey@0"
I20260929 18:44:45.407298 21132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:45.425836 21132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:45.428802 21132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:45.428984 21132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:45.435588 21132 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:39303
I20260929 18:44:45.435600 21199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:39303 every 8 connection(s)
I20260929 18:44:45.436523 21132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:44:45.439289 21200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:44:45.443045 21200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:44:45.445274 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 21132
I20260929 18:44:45.445385 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:44:45.445647 7797 external_mini_cluster.cc:1428] Setting key 8ab11d79b797df2a0000a39236985bdc
WARNING: no policy specified for kudu/127.7.157.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.125@KRBTEST.COM" created.
Sep 29 18:44:45 dist-test-slave-smpn krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:45.460830 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.445384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.7.157.126:39303)
0929 18:44:45.445898 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:45.445918 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:45.446196 (+ 278us) server_negotiation.cc:407] Connection header received
0929 18:44:45.447238 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:45.447242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:45.447432 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:45.447600 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:45.448699 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.450093 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.451125 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.451703 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.451869 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.456251 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:45.456258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:45.456273 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:45.456299 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:45.457873 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.458381 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.458385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.458386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.458483 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.458783 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.458785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.458786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.459240 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:45.459341 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:45.459775 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 18:44:45.460018 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":341,"spinlock_wait_cycles":896,"thread_start_us":157,"threads_started":1}
I20260929 18:44:45.460837 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.445384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39303 (local address 127.0.0.1:60574)
0929 18:44:45.446030 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.446079 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:45.446910 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:45.447610 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:45.447616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:45.448051 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:45.448307 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.448322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.450223 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.450226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.450990 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.451000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.451853 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.451855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.451945 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.452502 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:45.452519 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:45.456091 (+ 3572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:45.457988 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.457997 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.458008 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.458288 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.458559 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.458561 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.458563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.458700 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.459348 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:45.459356 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:45.459534 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 18:44:45.459732 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 18:44:45.460096 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":58,"spinlock_wait_cycles":55680,"thread_start_us":157,"threads_started":1}
I20260929 18:44:45.463358 21200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:44:45.464031 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.463792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42965 (local address 127.0.0.1:34398)
0929 18:44:45.463897 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.463967 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:42965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 18:44:45.464238 21151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.157.125:42965: connect: Connection refused (error 111)
I20260929 18:44:45.465152 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.464880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42965 (local address 127.0.0.1:34402)
0929 18:44:45.465042 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.465093 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:42965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 18:44:45.465332 21200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.125:42965: Network error: Client connection negotiation failed: client connection to 127.7.157.125:42965: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.125.keytab.
Entry for principal kudu/127.7.157.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.125.keytab.
I20260929 18:44:45.469945 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.125:42965
--webserver_interface=127.7.157.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.125.keytab
--principal=kudu/127.7.157.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:39303,127.7.157.125:42965,127.7.157.124:41413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:44:45.514559 21200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } attempt: 1
I20260929 18:44:45.515424 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.515118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42965 (local address 127.0.0.1:34410)
0929 18:44:45.515277 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.515342 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:42965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 18:44:45.516108 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.515897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42965 (local address 127.0.0.1:34422)
0929 18:44:45.515999 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.516045 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:42965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 18:44:45.516229 21200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.125:42965: Network error: Client connection negotiation failed: client connection to 127.7.157.125:42965: connect: Connection refused (error 111)
W20260929 18:44:45.577904 21207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:45.578168 21207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:45.578224 21207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:45.582214 21207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:44:45.582269 21207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:45.582295 21207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:45.582315 21207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:44:45.582341 21207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:44:45.584405 21200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } attempt: 2
I20260929 18:44:45.585196 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.584911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42965 (local address 127.0.0.1:34424)
0929 18:44:45.585058 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.585123 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:42965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 18:44:45.585876 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.585663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42965 (local address 127.0.0.1:34434)
0929 18:44:45.585771 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.585817 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:42965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 18:44:45.586056 21200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.125:42965: Network error: Client connection negotiation failed: client connection to 127.7.157.125:42965: connect: Connection refused (error 111)
I20260929 18:44:45.587607 21207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:39303,127.7.157.125:42965,127.7.157.124:41413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.125.keytab
--principal=kudu/127.7.157.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.125:42965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.157.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:45.588601 21207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:45.589545 21207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:45.597744 21213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:45.597712 21212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:45.597705 21215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:45.597904 21207 server_base.cc:1034] running on GCE node
I20260929 18:44:45.598569 21207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:45.599721 21207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:45.600875 21207 hybrid_clock.cc:648] HybridClock initialized: now 1790707485600857 us; error 34 us; skew 500 ppm
Sep 29 18:44:45 dist-test-slave-smpn krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:45.603479 21207 init.cc:377] Logged in from keytab as kudu/127.7.157.125@KRBTEST.COM (short username kudu)
I20260929 18:44:45.604358 21207 webserver.cc:460] Webserver started at http://127.7.157.125:39831/ using document root <none> and password file <none>
I20260929 18:44:45.604933 21207 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:45.604997 21207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:45.605183 21207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:44:45.607333 21207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/data/instance:
uuid: "8fede8e0dd1444558ead842941ad5545"
format_stamp: "Formatted at 2026-09-29 18:44:45 on dist-test-slave-smpn"
server_key: "902f871a35b1d3a935e23d617ac61999"
server_key_iv: "db7fbc486385a3047a98cea09e156ccd"
server_key_version: "encryptionkey@0"
I20260929 18:44:45.607869 21207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/wal/instance:
uuid: "8fede8e0dd1444558ead842941ad5545"
format_stamp: "Formatted at 2026-09-29 18:44:45 on dist-test-slave-smpn"
server_key: "902f871a35b1d3a935e23d617ac61999"
server_key_iv: "db7fbc486385a3047a98cea09e156ccd"
server_key_version: "encryptionkey@0"
I20260929 18:44:45.611610 21207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:44:45.614001 21222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:45.615226 21207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:44:45.615331 21207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/wal
uuid: "8fede8e0dd1444558ead842941ad5545"
format_stamp: "Formatted at 2026-09-29 18:44:45 on dist-test-slave-smpn"
server_key: "902f871a35b1d3a935e23d617ac61999"
server_key_iv: "db7fbc486385a3047a98cea09e156ccd"
server_key_version: "encryptionkey@0"
I20260929 18:44:45.615422 21207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:45.625119 21207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:45.627848 21207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:45.628018 21207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:45.634421 21207 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.125:42965
I20260929 18:44:45.634460 21274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.125:42965 every 8 connection(s)
I20260929 18:44:45.635353 21207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/data/info.pb
I20260929 18:44:45.637956 21275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:44:45.641570 21275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:44:45.645553 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 21207
I20260929 18:44:45.645651 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-1/wal/instance
I20260929 18:44:45.645892 7797 external_mini_cluster.cc:1428] Setting key ba05ad301f9bf9831fc8174b50ec33b3
Sep 29 18:44:45 dist-test-slave-smpn krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.125@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.157.124@KRBTEST.COM; defaulting to no policy
I20260929 18:44:45.654471 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.643355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.7.157.126:39303)
0929 18:44:45.643497 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:45.643500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:45.644268 (+ 768us) server_negotiation.cc:407] Connection header received
0929 18:44:45.645179 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:45.645183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:45.645245 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:45.645334 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:45.646533 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.647127 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.648277 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.648529 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.648627 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.651187 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:45.651190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:45.651192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:45.651218 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:45.652500 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.653064 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.653066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.653067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.653132 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.653473 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.653476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.653477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.653654 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:45.653727 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:45.654215 (+ 488us) server_negotiation.cc:299] Negotiation successful
0929 18:44:45.654336 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:44:45.655354 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.643715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39303 (local address 127.0.0.1:60578)
0929 18:44:45.644120 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.644166 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:45.644961 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:45.645503 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:45.645510 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:45.645881 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:45.646388 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.646402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.647274 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.647277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.648161 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.648170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.648701 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.648706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.648879 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.649448 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:45.649464 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:45.650978 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:45.652615 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.652621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.652636 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.652970 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.653214 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.653217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.653219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.653394 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.653740 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:45.653745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:45.653990 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 18:44:45.654164 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 18:44:45.654433 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.124@KRBTEST.COM" created.
I20260929 18:44:45.656745 21275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:44:45 dist-test-slave-smpn krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.125@KRBTEST.COM for kudu/127.7.157.125@KRBTEST.COM
I20260929 18:44:45.665184 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.657231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42965 (local address 127.0.0.1:34446)
0929 18:44:45.657567 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.657578 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:45.657640 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:45.658075 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:45.658078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:45.658247 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:45.658348 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.658355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.659334 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.659337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.659864 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.659871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.660414 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.660418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.660525 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.660993 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:45.661004 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:45.662195 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:45.663866 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.663869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.663870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.664068 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.664311 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.664313 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.664314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.664383 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.664773 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:45.664776 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:45.664870 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 18:44:45.664954 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:44:45.665056 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":3}
I20260929 18:44:45.665355 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.657314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.7.157.125:42965)
0929 18:44:45.657596 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:45.657607 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:45.657630 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:44:45.657706 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:45.657712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:45.657921 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:45.658059 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:45.658447 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.659228 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.659959 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.660315 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.660403 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.662326 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:45.662334 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:45.662346 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:45.662366 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:45.663775 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.664165 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.664169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.664171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.664243 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.664443 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.664445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.664446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.664680 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:45.664778 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:45.665025 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 18:44:45.665137 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
I20260929 18:44:45.667045 21275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:44:45.667702 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.667481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:41413 (local address 127.0.0.1:51830)
0929 18:44:45.667580 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.667647 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 18:44:45.667865 21225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
I20260929 18:44:45.668668 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.668471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:41413 (local address 127.0.0.1:51832)
0929 18:44:45.668561 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.668610 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 18:44:45.668780 21275 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:41413: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.124.keytab.
Entry for principal kudu/127.7.157.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.124.keytab.
I20260929 18:44:45.670166 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.124:41413
--webserver_interface=127.7.157.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.124.keytab
--principal=kudu/127.7.157.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:39303,127.7.157.125:42965,127.7.157.124:41413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:44:45.677210 21200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } attempt: 3
Sep 29 18:44:45 dist-test-slave-smpn krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for kudu/127.7.157.125@KRBTEST.COM
I20260929 18:44:45.685868 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.677698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42965 (local address 127.0.0.1:34452)
0929 18:44:45.677824 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.677840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:45.677942 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:45.678176 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:45.678179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:45.678347 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:45.678790 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.678799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.679948 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.679952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.680510 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.680521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.680970 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.680974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.681084 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.681530 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:45.681543 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:45.682908 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:45.684500 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.684503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.684505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.684701 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.684938 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.684941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.684943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.685022 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.685361 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:45.685364 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:45.685460 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 18:44:45.685551 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:44:45.685690 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 18:44:45.685866 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.677786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.7.157.125:42965)
0929 18:44:45.677925 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:45.677929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:45.677945 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:44:45.678022 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:45.678025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:45.678072 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:45.678137 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:45.678919 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.679835 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.680620 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.680865 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.680952 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.683087 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:45.683092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:45.683095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:45.683122 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:45.684397 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.684785 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.684789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.684790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.684860 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.685098 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.685101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.685103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.685276 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:45.685361 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:45.685615 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 18:44:45.685723 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 18:44:45.686597 21200 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:44:45.687367 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.687133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:41413 (local address 127.0.0.1:55472)
0929 18:44:45.687253 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.687303 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 18:44:45.688019 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.687778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:41413 (local address 127.0.0.1:55476)
0929 18:44:45.687921 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.687965 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260929 18:44:45.688179 21200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:41413: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
I20260929 18:44:45.718087 21275 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } attempt: 1
I20260929 18:44:45.719043 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.718682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:41413 (local address 127.0.0.1:55488)
0929 18:44:45.718890 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.718953 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260929 18:44:45.719421 21200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } attempt: 1
I20260929 18:44:45.719991 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.719733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:41413 (local address 127.0.0.1:55500)
0929 18:44:45.719851 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.719916 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 18:44:45.720072 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.719785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:41413 (local address 127.0.0.1:55516)
0929 18:44:45.719912 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.719964 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 18:44:45.720280 21275 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:41413: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
I20260929 18:44:45.720719 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.720516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:41413 (local address 127.0.0.1:55530)
0929 18:44:45.720620 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.720666 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 18:44:45.720854 21200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:41413: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
W20260929 18:44:45.778941 21282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:45.779253 21282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:45.779301 21282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:45.783284 21282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:44:45.783337 21282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:45.783355 21282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:45.783373 21282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:44:45.783401 21282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:44:45.788462 21275 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } attempt: 2
I20260929 18:44:45.788738 21282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:39303,127.7.157.125:42965,127.7.157.124:41413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.124.keytab
--principal=kudu/127.7.157.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.124:41413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.157.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:45.789306 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.788977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:41413 (local address 127.0.0.1:55536)
0929 18:44:45.789144 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.789216 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 18:44:45.789973 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.789767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:41413 (local address 127.0.0.1:55546)
0929 18:44:45.789860 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.789911 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 18:44:45.790066 21282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260929 18:44:45.790112 21275 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:41413: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
I20260929 18:44:45.791106 21282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:44:45.795993 21200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } attempt: 2
I20260929 18:44:45.796639 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.796389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:41413 (local address 127.0.0.1:55550)
0929 18:44:45.796510 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.796570 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 18:44:45.797290 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.797107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:41413 (local address 127.0.0.1:55562)
0929 18:44:45.797197 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.797238 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 18:44:45.797392 21200 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:41413: Network error: Client connection negotiation failed: client connection to 127.7.157.124:41413: connect: Connection refused (error 111)
W20260929 18:44:45.799636 21290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:45.799657 21287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:45.799713 21282 server_base.cc:1034] running on GCE node
W20260929 18:44:45.799777 21288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:45.800505 21282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:45.801553 21282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:45.802711 21282 hybrid_clock.cc:648] HybridClock initialized: now 1790707485802689 us; error 31 us; skew 500 ppm
Sep 29 18:44:45 dist-test-slave-smpn krb5kdc[21119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:45.805289 21282 init.cc:377] Logged in from keytab as kudu/127.7.157.124@KRBTEST.COM (short username kudu)
I20260929 18:44:45.806150 21282 webserver.cc:460] Webserver started at http://127.7.157.124:41925/ using document root <none> and password file <none>
I20260929 18:44:45.806686 21282 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:45.806749 21282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:45.806931 21282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:44:45.808923 21282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/data/instance:
uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed"
format_stamp: "Formatted at 2026-09-29 18:44:45 on dist-test-slave-smpn"
server_key: "6d9305e50aa933000ab36f7e82a9616d"
server_key_iv: "0f5a9ac89c4be286ecb73bbbd2a65ab5"
server_key_version: "encryptionkey@0"
I20260929 18:44:45.809437 21282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/wal/instance:
uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed"
format_stamp: "Formatted at 2026-09-29 18:44:45 on dist-test-slave-smpn"
server_key: "6d9305e50aa933000ab36f7e82a9616d"
server_key_iv: "0f5a9ac89c4be286ecb73bbbd2a65ab5"
server_key_version: "encryptionkey@0"
I20260929 18:44:45.813053 21282 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 18:44:45.815361 21297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:45.816541 21282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:44:45.816663 21282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/wal
uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed"
format_stamp: "Formatted at 2026-09-29 18:44:45 on dist-test-slave-smpn"
server_key: "6d9305e50aa933000ab36f7e82a9616d"
server_key_iv: "0f5a9ac89c4be286ecb73bbbd2a65ab5"
server_key_version: "encryptionkey@0"
I20260929 18:44:45.816751 21282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:45.852540 21282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:45.855382 21282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:45.855559 21282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:45.862280 21282 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.124:41413
I20260929 18:44:45.862303 21349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.124:41413 every 8 connection(s)
I20260929 18:44:45.863268 21282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/data/info.pb
I20260929 18:44:45.866020 21350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:44:45.866298 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 21282
I20260929 18:44:45.866407 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707328618849-7797-0/minicluster-data/master-2/wal/instance
I20260929 18:44:45.866724 7797 external_mini_cluster.cc:1428] Setting key 47b92fcf2083192a20994554a8834b47
I20260929 18:44:45.870541 21350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:44:45 dist-test-slave-smpn krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.124@KRBTEST.COM
Sep 29 18:44:45 dist-test-slave-smpn krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.124@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:45.880535 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.868104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.7.157.124:41413)
0929 18:44:45.868514 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:45.868528 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:45.868563 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:44:45.869238 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:45.869256 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:45.869577 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:45.869880 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:45.870642 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.871709 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.872632 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.873214 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.873356 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.875780 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:45.875785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:45.875794 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:45.875819 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:45.877312 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.877734 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.877740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.877746 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.877868 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.878176 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.878179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.878181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.878680 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:45.878868 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:45.879104 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 18:44:45.879563 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":273,"thread_start_us":125,"threads_started":1}
I20260929 18:44:45.880792 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.872229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45782 (local address 127.7.157.126:39303)
0929 18:44:45.872364 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:45.872369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:45.872635 (+ 266us) server_negotiation.cc:407] Connection header received
0929 18:44:45.872821 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:45.872824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:45.872874 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:45.872954 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:45.873615 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.874081 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.875027 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.875326 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.875418 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.877653 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:45.877656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:45.877658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:45.877681 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:45.879186 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.879695 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.879698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.879699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.879772 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.880012 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.880014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.880015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.880181 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:45.880251 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:45.880549 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 18:44:45.880653 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:44:45.880853 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.872134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39303 (local address 127.0.0.1:45782)
0929 18:44:45.872514 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.872561 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:45.872716 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:45.873018 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:45.873022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:45.873330 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:45.873503 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.873509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.874188 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.874190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.874901 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.874911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.875507 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.875512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.875624 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.876150 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:45.876161 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:45.877517 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:45.879302 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.879309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.879314 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.879601 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.879851 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.879854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.879855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.879941 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.880267 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:45.880273 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:45.880382 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 18:44:45.880516 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 18:44:45.880685 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":20,"thread_start_us":125,"threads_started":1}
I20260929 18:44:45.881268 21275 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } attempt: 3
I20260929 18:44:45.882166 21350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:44:45 dist-test-slave-smpn krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.125@KRBTEST.COM for kudu/127.7.157.124@KRBTEST.COM
Sep 29 18:44:45 dist-test-slave-smpn krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.124@KRBTEST.COM for kudu/127.7.157.125@KRBTEST.COM
Sep 29 18:44:45 dist-test-slave-smpn krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:45.890730 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.881737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55584 (local address 127.7.157.124:41413)
0929 18:44:45.881822 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:45.881825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:45.881879 (+ 54us) server_negotiation.cc:407] Connection header received
0929 18:44:45.881999 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:45.882001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:45.882039 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:45.882555 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:45.882631 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.883353 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.884092 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.884423 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.884516 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.886563 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:45.886567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:45.886570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:45.886594 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:45.889154 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.889652 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.889656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.889657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.889731 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.889978 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.889981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.889983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.890136 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:45.890215 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:45.890490 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 18:44:45.890583 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:44:45.890734 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.881706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:41413 (local address 127.0.0.1:55584)
0929 18:44:45.881825 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.881839 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:45.881925 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:45.882163 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:45.882166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:45.882337 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:45.882444 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.882452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.883450 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.883454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.883972 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.883979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.884530 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.884533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.884616 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.885117 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:45.885131 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:45.886423 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:45.889306 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.889310 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.889312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.889562 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.889808 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.889812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.889814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.889895 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.890219 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:45.890222 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:45.890309 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 18:44:45.890429 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 18:44:45.890584 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":2}
I20260929 18:44:45.891619 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.882796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38398 (local address 127.7.157.125:42965)
0929 18:44:45.882890 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:45.882894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:45.882905 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:44:45.882934 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:45.882935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:45.882974 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:45.883039 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:45.884186 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.884731 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.885672 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.885964 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.886049 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.887929 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:45.887932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:45.887935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:45.887958 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:45.889491 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.889944 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.889948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.889950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.890028 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.890328 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.890332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.890334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.891009 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:45.891129 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:45.891372 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 18:44:45.891476 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":452,"server-negotiator.queue_time_us":30}
I20260929 18:44:45.891809 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.883132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.7.157.126:39303)
0929 18:44:45.883242 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:45.883245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:45.883256 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:44:45.883282 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:45.883284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:45.883312 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:45.883369 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:45.884048 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.884706 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.885513 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.885773 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.885862 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.888585 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:45.888589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:45.888591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:45.888614 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:45.890063 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.890489 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.890493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.890494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.890575 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.890981 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.890984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.890986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.891204 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:45.891429 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:45.891575 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 18:44:45.891673 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 18:44:45.892416 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.882591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:42965 (local address 127.0.0.1:38398)
0929 18:44:45.882676 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.882689 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:45.882753 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:45.883442 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:45.883445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:45.883601 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:45.884083 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.884093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.884847 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.884850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.885562 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.885572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.886063 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.886066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.886169 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.886510 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:45.886522 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:45.887787 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:45.889618 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.889621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.889623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.889831 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.890136 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.890138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.890140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.890225 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.891112 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:45.891115 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:45.891195 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 18:44:45.892148 (+ 953us) client_negotiation.cc:240] Negotiation successful
0929 18:44:45.892270 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 18:44:45.893812 21275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545: Bootstrap starting.
I20260929 18:44:45.895738 21350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:44:45.896417 21275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545: Neither blocks nor log segments found. Creating new log.
I20260929 18:44:45.896528 21200 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } attempt: 3
Sep 29 18:44:45 dist-test-slave-smpn krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.125@KRBTEST.COM
I20260929 18:44:45.897323 21275 log.cc:824] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:45.899791 21275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545: No bootstrap required, opened a new log
I20260929 18:44:45.901233 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.892877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38400 (local address 127.7.157.125:42965)
0929 18:44:45.892975 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:45.892979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:45.892991 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:44:45.893064 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:45.893067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:45.893102 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:45.893165 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:45.893702 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.894221 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.894924 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.895229 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.895317 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.898043 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:45.898047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:45.898049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:45.898072 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:45.899514 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.899948 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.899951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.899953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.900030 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.900317 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.900320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.900322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.900525 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:45.900665 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:45.900998 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 18:44:45.901099 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 29 18:44:45 dist-test-slave-smpn krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.124@KRBTEST.COM for kudu/127.7.157.124@KRBTEST.COM
Sep 29 18:44:45 dist-test-slave-smpn krb5kdc[21119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707485, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for kudu/127.7.157.124@KRBTEST.COM
I20260929 18:44:45.904224 21275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } }
I20260929 18:44:45.904453 21275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:44:45.904528 21275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fede8e0dd1444558ead842941ad5545, State: Initialized, Role: FOLLOWER
I20260929 18:44:45.905046 21275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } }
I20260929 18:44:45.905610 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:44:45.905578 21357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } } }
I20260929 18:44:45.905695 21357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:44:45.906500 21275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:44:45.906793 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.896873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:41413 (local address 127.0.0.1:55592)
0929 18:44:45.896964 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.896977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:45.897041 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:45.898006 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:45.898009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:45.898172 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:45.898270 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.898278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.900026 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.900029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.900786 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.900797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.901351 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.901354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.901436 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.901842 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:45.901856 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:45.902945 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:45.905152 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.905155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.905157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.905398 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.905823 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.905826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.905828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.905907 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.906295 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:45.906298 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:45.906396 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 18:44:45.906512 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 18:44:45.906644 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 18:44:45.907339 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.897383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.7.157.124:41413)
0929 18:44:45.897771 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:45.897775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:45.897789 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:44:45.897820 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:45.897823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:45.897860 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:45.899134 (+ 1274us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:45.899205 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.899917 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.900925 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.901222 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.901640 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.903104 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:45.903108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:45.903110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:45.903134 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:45.905034 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.905651 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.905654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.905656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.905731 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.906002 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.906004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.906006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.906197 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:45.906285 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:45.906587 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 18:44:45.907194 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":131,"threads_started":1}
I20260929 18:44:45.909482 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.896191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:41413 (local address 127.0.0.1:55586)
0929 18:44:45.896300 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:45.896313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:45.896390 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:45.897902 (+ 1512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:45.897904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:45.898052 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:45.898140 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.898147 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.899061 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.899100 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.899887 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:45.899895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.900478 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.900481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:45.900554 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.900953 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:45.900966 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:45.903669 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:45.908147 (+ 4478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.908150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.908152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.908371 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.908649 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:45.908652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:45.908653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:45.908737 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:45.909121 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:45.909124 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:45.909206 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 18:44:45.909281 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:44:45.909379 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":4}
I20260929 18:44:45.909893 21355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.896396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.7.157.124:41413)
0929 18:44:45.897691 (+ 1295us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:45.897700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:45.897714 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:44:45.897746 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:45.897748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:45.897786 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:45.898220 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:45.898295 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.898949 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.900004 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.900372 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.903785 (+ 3413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.904089 (+ 304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:45.904095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:45.904098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:45.904123 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:45.908037 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.908467 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.908470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.908472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.908563 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.908831 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.908833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.908835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.909025 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:45.909640 (+ 615us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:45.909707 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 18:44:45.909800 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1233}
I20260929 18:44:45.910283 21200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f: Bootstrap starting.
W20260929 18:44:45.912170 21373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:44:45.912242 21373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:44:45.913937 21350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed: Bootstrap starting.
I20260929 18:44:45.916616 21350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed: Neither blocks nor log segments found. Creating new log.
I20260929 18:44:45.917500 21350 log.cc:824] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:45.921452 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.908438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.7.157.124:41413)
0929 18:44:45.908531 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:45.908535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:45.913007 (+ 4472us) server_negotiation.cc:407] Connection header received
0929 18:44:45.913163 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:45.913166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:45.913202 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:45.913245 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:45.914051 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.914721 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.915906 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.916211 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.916280 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.918034 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:45.918037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:45.918040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:45.918063 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:45.919670 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.920181 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.920185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.920187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.920265 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.920542 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.920545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.920547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.920747 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:45.920844 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:45.921200 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 18:44:45.921308 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:44:45.921825 21200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f: Neither blocks nor log segments found. Creating new log.
I20260929 18:44:45.919041 21207 master_runner.cc:186] Error getting master registration for 127.7.157.126:39303: OK, instance_id { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" instance_seqno: 1790707485433722 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:44:45.922765 21200 log.cc:824] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:45.923311 21207 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:44:45.925206 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.907822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45800 (local address 127.7.157.126:39303)
0929 18:44:45.907935 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:45.907939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:45.912640 (+ 4701us) server_negotiation.cc:407] Connection header received
0929 18:44:45.912764 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:45.912767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:45.912816 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:45.912887 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:45.913774 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.914519 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.915858 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.916188 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.916265 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.918166 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:45.918170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:45.918172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:45.918199 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:45.923029 (+ 4830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.923820 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.923824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.923826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.923915 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.924227 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.924231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.924233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.924442 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:45.924560 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:45.924922 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 18:44:45.925047 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:44:45.925228 21350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed: No bootstrap required, opened a new log
I20260929 18:44:45.929059 21200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f: No bootstrap required, opened a new log
I20260929 18:44:45.929760 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:45.909042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.7.157.125:42965)
0929 18:44:45.909141 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:45.909145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:45.921941 (+ 12796us) server_negotiation.cc:407] Connection header received
0929 18:44:45.922138 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:45.922141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:45.922189 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:45.922237 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:45.922953 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.923671 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.924673 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:45.924983 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:45.925058 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:45.926557 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:45.926560 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:45.926563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:45.926589 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:45.928148 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.928587 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.928590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.928592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.928676 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:45.928934 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:45.928937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:45.928938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:45.929156 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:45.929274 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:45.929506 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 18:44:45.929617 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:44:45.932773 21200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } }
I20260929 18:44:45.932778 21350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } }
I20260929 18:44:45.933001 21200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:44:45.933001 21350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:44:45.933058 21350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e67bb4430ef4b1d95c30e5bf0e41aed, State: Initialized, Role: FOLLOWER
I20260929 18:44:45.933086 21200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d29c8d68f147d593061cb3a45a637f, State: Initialized, Role: FOLLOWER
I20260929 18:44:45.933563 21200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } }
I20260929 18:44:45.933564 21350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } }
I20260929 18:44:45.934079 21379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } } }
I20260929 18:44:45.934265 21379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:44:45.934527 21380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } } }
I20260929 18:44:45.934711 21380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:44:45.934937 21200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:44:45.934937 21350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:44:45.940888 21394 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:44:45.941028 21394 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 18:44:45.942103 21403 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:44:45.942174 21403 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:44:45.944293 21132 master_runner.cc:263] No leader master found from master 72d29c8d68f147d593061cb3a45a637f
I20260929 18:44:45.944356 21132 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:44:45.945063 21282 master_runner.cc:263] No leader master found from master 9e67bb4430ef4b1d95c30e5bf0e41aed
I20260929 18:44:45.945120 21282 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:44:46.100942 21380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:44:46.101164 21380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } }
I20260929 18:44:46.102315 21380 leader_election.cc:288] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fede8e0dd1444558ead842941ad5545 (127.7.157.125:42965), 9e67bb4430ef4b1d95c30e5bf0e41aed (127.7.157.124:41413)
I20260929 18:44:46.102999 21249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72d29c8d68f147d593061cb3a45a637f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fede8e0dd1444558ead842941ad5545" is_pre_election: true
I20260929 18:44:46.103411 21249 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72d29c8d68f147d593061cb3a45a637f in term 0.
I20260929 18:44:46.103963 21151 leader_election.cc:302] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72d29c8d68f147d593061cb3a45a637f, 8fede8e0dd1444558ead842941ad5545; no voters:
I20260929 18:44:46.104748 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72d29c8d68f147d593061cb3a45a637f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" is_pre_election: true
I20260929 18:44:46.104805 21380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:44:46.104914 21380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:44:46.104962 21380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:44:46.105017 21324 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72d29c8d68f147d593061cb3a45a637f in term 0.
I20260929 18:44:46.106855 21380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } }
W20260929 18:44:46.106948 21303 tablet.cc:2369] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:44:46.107309 21380 leader_election.cc:288] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [CANDIDATE]: Term 1 election: Requested vote from peers 8fede8e0dd1444558ead842941ad5545 (127.7.157.125:42965), 9e67bb4430ef4b1d95c30e5bf0e41aed (127.7.157.124:41413)
I20260929 18:44:46.107870 21249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72d29c8d68f147d593061cb3a45a637f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fede8e0dd1444558ead842941ad5545"
I20260929 18:44:46.107980 21324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72d29c8d68f147d593061cb3a45a637f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed"
I20260929 18:44:46.107991 21249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:44:46.108117 21324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:44:46.109208 21249 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72d29c8d68f147d593061cb3a45a637f in term 1.
I20260929 18:44:46.109429 21324 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72d29c8d68f147d593061cb3a45a637f in term 1.
I20260929 18:44:46.109706 21151 leader_election.cc:302] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72d29c8d68f147d593061cb3a45a637f, 8fede8e0dd1444558ead842941ad5545; no voters:
I20260929 18:44:46.109931 21380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:44:46.110177 21380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [term 1 LEADER]: Becoming Leader. State: Replica: 72d29c8d68f147d593061cb3a45a637f, State: Running, Role: LEADER
I20260929 18:44:46.110570 21380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } }
I20260929 18:44:46.112726 21405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72d29c8d68f147d593061cb3a45a637f. Latest consensus state: current_term: 1 leader_uuid: "72d29c8d68f147d593061cb3a45a637f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } } }
I20260929 18:44:46.112895 21405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [sys.catalog]: This master's current role is: LEADER
I20260929 18:44:46.113287 21407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:44:46.114272 21407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:44:46.118861 21324 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed [term 1 FOLLOWER]: Refusing update from remote peer 72d29c8d68f147d593061cb3a45a637f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:44:46.118893 21249 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545 [term 1 FOLLOWER]: Refusing update from remote peer 72d29c8d68f147d593061cb3a45a637f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:44:46.119581 21405 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:44:46.119738 21380 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:44:46.125741 21379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72d29c8d68f147d593061cb3a45a637f. Latest consensus state: current_term: 1 leader_uuid: "72d29c8d68f147d593061cb3a45a637f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } } }
I20260929 18:44:46.125842 21379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:44:46.126991 21357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72d29c8d68f147d593061cb3a45a637f. Latest consensus state: current_term: 1 leader_uuid: "72d29c8d68f147d593061cb3a45a637f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } } }
I20260929 18:44:46.127128 21357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:44:46.128057 21405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72d29c8d68f147d593061cb3a45a637f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } } }
I20260929 18:44:46.128183 21405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [sys.catalog]: This master's current role is: LEADER
I20260929 18:44:46.128300 21408 mvcc.cc:205] Tried to move back new op lower bound from 7334737863136161792 to 7334737863112138752. Current Snapshot: MvccSnapshot[applied={T|T < 7334737863136161792}]
I20260929 18:44:46.128545 21409 mvcc.cc:205] Tried to move back new op lower bound from 7334737863136161792 to 7334737863112138752. Current Snapshot: MvccSnapshot[applied={T|T < 7334737863136161792}]
I20260929 18:44:46.128686 21379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72d29c8d68f147d593061cb3a45a637f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } } }
I20260929 18:44:46.128782 21379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:44:46.129251 21380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72d29c8d68f147d593061cb3a45a637f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } } }
I20260929 18:44:46.129359 21380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f [sys.catalog]: This master's current role is: LEADER
W20260929 18:44:46.129415 21228 tablet.cc:2369] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:44:46.130213 21357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72d29c8d68f147d593061cb3a45a637f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d29c8d68f147d593061cb3a45a637f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39303 } } peers { permanent_uuid: "8fede8e0dd1444558ead842941ad5545" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 42965 } } peers { permanent_uuid: "9e67bb4430ef4b1d95c30e5bf0e41aed" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 41413 } } }
I20260929 18:44:46.130313 21357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:44:46.130650 21411 mvcc.cc:205] Tried to move back new op lower bound from 7334737863136161792 to 7334737863112138752. Current Snapshot: MvccSnapshot[applied={T|T < 7334737863136161792}]
I20260929 18:44:46.132189 21407 catalog_manager.cc:1347] Generated new cluster ID: 0807c5c7fc8f48d09de8bf37e091963d
I20260929 18:44:46.132247 21407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:44:46.142234 21407 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:44:46.143442 21407 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:44:46.150147 21407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72d29c8d68f147d593061cb3a45a637f: Generated new TSK 0
I20260929 18:44:46.150880 21407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:44:46.913528 21373 catalog_manager.cc:1259] Loaded cluster ID: 0807c5c7fc8f48d09de8bf37e091963d
I20260929 18:44:46.913612 21373 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545: loading cluster ID for follower catalog manager: success
I20260929 18:44:46.915293 21373 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545: acquiring CA information for follower catalog manager: success
I20260929 18:44:46.916441 21373 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8fede8e0dd1444558ead842941ad5545: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:44:46.943423 21403 catalog_manager.cc:1259] Loaded cluster ID: 0807c5c7fc8f48d09de8bf37e091963d
I20260929 18:44:46.943503 21403 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed: loading cluster ID for follower catalog manager: success
I20260929 18:44:46.945151 21403 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed: acquiring CA information for follower catalog manager: success
I20260929 18:44:46.946346 21403 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9e67bb4430ef4b1d95c30e5bf0e41aed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:44:47.215615 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 21132
I20260929 18:44:47.220992 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 21207
I20260929 18:44:47.225895 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 21282
2026-09-29T18:44:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3039 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:44:47 dist-test-slave-smpn krb5kdc[21422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:44:47 dist-test-slave-smpn krb5kdc[21422](info): set up 2 sockets
Sep 29 18:44:47 dist-test-slave-smpn krb5kdc[21422](info): commencing operation
krb5kdc: starting...
W20260929 18:44:48.179234 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:44:48 dist-test-slave-smpn krb5kdc[21422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:44:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:44:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:44:48.291723 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:34837
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:34837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:44:48.401309 21435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:48.401574 21435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:48.401659 21435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:48.405648 21435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:44:48.405711 21435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:48.405737 21435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:48.405758 21435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:44:48.405783 21435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:44:48.411182 21435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:34837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:34837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:48.412307 21435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:48.413317 21435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:48.421566 21441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:48.421559 21440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:48.421697 21435 server_base.cc:1034] running on GCE node
W20260929 18:44:48.421576 21443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:48.422428 21435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:48.423693 21435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:48.424863 21435 hybrid_clock.cc:648] HybridClock initialized: now 1790707488424848 us; error 27 us; skew 500 ppm
Sep 29 18:44:48 dist-test-slave-smpn krb5kdc[21422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707488, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:48.427627 21435 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:44:48.428536 21435 webserver.cc:460] Webserver started at http://127.7.157.126:40507/ using document root <none> and password file <none>
I20260929 18:44:48.429100 21435 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:48.429164 21435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:48.429368 21435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:44:48.431502 21435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "f66c63e0d63141b5aa536d34476943fe"
format_stamp: "Formatted at 2026-09-29 18:44:48 on dist-test-slave-smpn"
server_key: "7fa8d38628cf72b9ba9602b100c0ef9e"
server_key_iv: "636dcbdd403defffe49c25806f5642e4"
server_key_version: "encryptionkey@0"
I20260929 18:44:48.432060 21435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "f66c63e0d63141b5aa536d34476943fe"
format_stamp: "Formatted at 2026-09-29 18:44:48 on dist-test-slave-smpn"
server_key: "7fa8d38628cf72b9ba9602b100c0ef9e"
server_key_iv: "636dcbdd403defffe49c25806f5642e4"
server_key_version: "encryptionkey@0"
I20260929 18:44:48.435868 21435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:44:48.438292 21450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:48.439589 21435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 18:44:48.439706 21435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "f66c63e0d63141b5aa536d34476943fe"
format_stamp: "Formatted at 2026-09-29 18:44:48 on dist-test-slave-smpn"
server_key: "7fa8d38628cf72b9ba9602b100c0ef9e"
server_key_iv: "636dcbdd403defffe49c25806f5642e4"
server_key_version: "encryptionkey@0"
I20260929 18:44:48.439812 21435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:48.459522 21435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:48.462316 21435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:48.462507 21435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:48.469412 21435 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:34837
I20260929 18:44:48.469439 21502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:34837 every 8 connection(s)
I20260929 18:44:48.470386 21435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:44:48.473186 21503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:44:48.478472 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 21435
I20260929 18:44:48.478612 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:44:48.478899 7797 external_mini_cluster.cc:1428] Setting key 5582f9ac02e5589390bc289b2aeac5b4
I20260929 18:44:48.479768 21503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe: Bootstrap starting.
I20260929 18:44:48.482228 21503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe: Neither blocks nor log segments found. Creating new log.
I20260929 18:44:48.482961 21503 log.cc:824] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:48.484807 21503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe: No bootstrap required, opened a new log
Sep 29 18:44:48 dist-test-slave-smpn krb5kdc[21422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:48.488754 21503 raft_consensus.cc:348] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66c63e0d63141b5aa536d34476943fe" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 34837 } }
I20260929 18:44:48.489012 21503 raft_consensus.cc:374] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:44:48.489073 21503 raft_consensus.cc:729] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66c63e0d63141b5aa536d34476943fe, State: Initialized, Role: FOLLOWER
I20260929 18:44:48.489522 21503 consensus_queue.cc:260] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66c63e0d63141b5aa536d34476943fe" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 34837 } }
I20260929 18:44:48.489658 21503 raft_consensus.cc:388] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:44:48.489723 21503 raft_consensus.cc:482] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:44:48.489820 21503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:44:48.490675 21503 raft_consensus.cc:504] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66c63e0d63141b5aa536d34476943fe" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 34837 } }
I20260929 18:44:48.490994 21503 leader_election.cc:302] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f66c63e0d63141b5aa536d34476943fe; no voters:
I20260929 18:44:48.491314 21503 leader_election.cc:288] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:44:48.491398 21508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:44:48.491596 21508 raft_consensus.cc:686] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [term 1 LEADER]: Becoming Leader. State: Replica: f66c63e0d63141b5aa536d34476943fe, State: Running, Role: LEADER
I20260929 18:44:48.491963 21508 consensus_queue.cc:237] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66c63e0d63141b5aa536d34476943fe" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 34837 } }
I20260929 18:44:48.492630 21503 sys_catalog.cc:564] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:44:48.492913 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:48.480332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.7.157.126:34837)
0929 18:44:48.480770 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:48.480785 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:48.480822 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:44:48.481457 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:48.481474 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:48.481836 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:48.482150 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:48.482942 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:48.483975 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:48.484824 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:48.485356 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:48.485502 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:48.488139 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:48.488144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:48.488157 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:48.488183 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:48.489817 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:48.490220 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:48.490226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:48.490232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:48.490346 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:48.490601 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:48.490605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:48.490606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:48.491047 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:48.491254 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:48.491785 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 18:44:48.492088 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":134,"threads_started":1}
I20260929 18:44:48.493148 21508 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader f66c63e0d63141b5aa536d34476943fe. Latest consensus state: current_term: 1 leader_uuid: "f66c63e0d63141b5aa536d34476943fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66c63e0d63141b5aa536d34476943fe" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 34837 } } }
I20260929 18:44:48.493892 21508 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [sys.catalog]: This master's current role is: LEADER
I20260929 18:44:48.493783 21509 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f66c63e0d63141b5aa536d34476943fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66c63e0d63141b5aa536d34476943fe" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 34837 } } }
I20260929 18:44:48.494192 21509 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe [sys.catalog]: This master's current role is: LEADER
I20260929 18:44:48.496116 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 18:44:48.497611 21523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:44:48.497690 21523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:44:48.497756 21517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:44:48.498420 21517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:44:48.504143 21517 catalog_manager.cc:1347] Generated new cluster ID: 7c4160c09a3c4b3a87b22c8e0b98c27a
I20260929 18:44:48.504221 21517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:44:48.528412 21517 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:44:48.529397 21517 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:44:48.535961 21517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f66c63e0d63141b5aa536d34476943fe: Generated new TSK 0
I20260929 18:44:48.536667 21517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:44:48.551925 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 21435
2026-09-29T18:44:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1325 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:44:48 dist-test-slave-smpn krb5kdc[21530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:44:48 dist-test-slave-smpn krb5kdc[21530](info): set up 2 sockets
Sep 29 18:44:48 dist-test-slave-smpn krb5kdc[21530](info): commencing operation
krb5kdc: starting...
W20260929 18:44:49.495263 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:44:49 dist-test-slave-smpn krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:44:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:44:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:44:49.606213 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:32875
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:32875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:44:49.712921 21543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:49.713202 21543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:49.713258 21543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:49.717250 21543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:44:49.717309 21543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:49.717334 21543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:49.717355 21543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:44:49.717374 21543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:44:49.722743 21543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:32875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:32875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:49.723836 21543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:49.724836 21543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:49.733095 21548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:49.733215 21543 server_base.cc:1034] running on GCE node
W20260929 18:44:49.733180 21549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:49.733099 21551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:49.733862 21543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:49.734936 21543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:49.736078 21543 hybrid_clock.cc:648] HybridClock initialized: now 1790707489736048 us; error 43 us; skew 500 ppm
Sep 29 18:44:49 dist-test-slave-smpn krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707489, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:49.738624 21543 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:44:49.739508 21543 webserver.cc:460] Webserver started at http://127.7.157.126:43827/ using document root <none> and password file <none>
I20260929 18:44:49.740022 21543 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:49.740085 21543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:49.740259 21543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:44:49.742198 21543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "cab739d0beb54c649c66754a35c9af16"
format_stamp: "Formatted at 2026-09-29 18:44:49 on dist-test-slave-smpn"
server_key: "dbc8063d9d42a95f3968773d7daf4a22"
server_key_iv: "d2ac2cf968083a6e9a966d6eb96f7c83"
server_key_version: "encryptionkey@0"
I20260929 18:44:49.742725 21543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "cab739d0beb54c649c66754a35c9af16"
format_stamp: "Formatted at 2026-09-29 18:44:49 on dist-test-slave-smpn"
server_key: "dbc8063d9d42a95f3968773d7daf4a22"
server_key_iv: "d2ac2cf968083a6e9a966d6eb96f7c83"
server_key_version: "encryptionkey@0"
I20260929 18:44:49.746297 21543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:44:49.748616 21558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:49.749773 21543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 18:44:49.749890 21543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "cab739d0beb54c649c66754a35c9af16"
format_stamp: "Formatted at 2026-09-29 18:44:49 on dist-test-slave-smpn"
server_key: "dbc8063d9d42a95f3968773d7daf4a22"
server_key_iv: "d2ac2cf968083a6e9a966d6eb96f7c83"
server_key_version: "encryptionkey@0"
I20260929 18:44:49.749984 21543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:49.760319 21543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:49.763273 21543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:49.763435 21543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:49.770081 21543 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:32875
I20260929 18:44:49.770332 21610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:32875 every 8 connection(s)
I20260929 18:44:49.771023 21543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:44:49.771912 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 21543
I20260929 18:44:49.772032 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:44:49.772323 7797 external_mini_cluster.cc:1428] Setting key f1e22c17b768837513425d1757856008
I20260929 18:44:49.774245 21611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:44:49.779505 21611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16: Bootstrap starting.
Sep 29 18:44:49 dist-test-slave-smpn krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:49.781633 21611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16: Neither blocks nor log segments found. Creating new log.
I20260929 18:44:49.782308 21611 log.cc:824] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:49.784086 21611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16: No bootstrap required, opened a new log
I20260929 18:44:49.785181 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:49.773695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46078 (local address 127.7.157.126:32875)
0929 18:44:49.774119 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:49.774132 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:49.774166 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:44:49.774815 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:49.774832 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:49.775199 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:49.775500 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:49.776246 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:49.777224 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:49.778049 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:49.778508 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:49.778657 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:49.780946 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:49.780951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:49.780963 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:49.780989 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:49.782575 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:49.782963 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:49.782967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:49.782972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:49.783118 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:49.783366 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:49.783369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:49.783371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:49.783756 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:49.783923 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:49.784206 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 18:44:49.784481 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":139,"threads_started":1}
I20260929 18:44:49.787235 21611 raft_consensus.cc:348] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } }
I20260929 18:44:49.787503 21611 raft_consensus.cc:374] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:44:49.787607 21611 raft_consensus.cc:729] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab739d0beb54c649c66754a35c9af16, State: Initialized, Role: FOLLOWER
I20260929 18:44:49.788017 21611 consensus_queue.cc:260] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } }
I20260929 18:44:49.788136 21611 raft_consensus.cc:388] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:44:49.788180 21611 raft_consensus.cc:482] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:44:49.788246 21611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:44:49.789131 21611 raft_consensus.cc:504] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } }
I20260929 18:44:49.789484 21611 leader_election.cc:302] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cab739d0beb54c649c66754a35c9af16; no voters:
I20260929 18:44:49.789774 21611 leader_election.cc:288] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:44:49.789870 21616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:44:49.790066 21616 raft_consensus.cc:686] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 1 LEADER]: Becoming Leader. State: Replica: cab739d0beb54c649c66754a35c9af16, State: Running, Role: LEADER
I20260929 18:44:49.790385 21616 consensus_queue.cc:237] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } }
I20260929 18:44:49.790839 21611 sys_catalog.cc:564] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:44:49.791981 21618 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cab739d0beb54c649c66754a35c9af16. Latest consensus state: current_term: 1 leader_uuid: "cab739d0beb54c649c66754a35c9af16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } } }
I20260929 18:44:49.791980 21617 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cab739d0beb54c649c66754a35c9af16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } } }
I20260929 18:44:49.792119 21618 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [sys.catalog]: This master's current role is: LEADER
I20260929 18:44:49.792119 21617 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [sys.catalog]: This master's current role is: LEADER
I20260929 18:44:49.792603 21626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:44:49.795390 21626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:44:49.800459 21626 catalog_manager.cc:1347] Generated new cluster ID: 9314cd147c3f404b895f695e4e02ccab
I20260929 18:44:49.800521 21626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:44:49.807513 21626 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:44:49.808396 21626 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:44:49.812060 21626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16: Generated new TSK 0
I20260929 18:44:49.812832 21626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:44:49.840365 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:32875
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:44:49.950495 21637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:49.950766 21637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:49.950827 21637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:49.954957 21637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:49.955026 21637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:49.955125 21637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:44:49.960379 21637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:49.961398 21637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:49.962400 21637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:49.971379 21643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:49.971388 21645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:49.971386 21642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:49.971701 21637 server_base.cc:1034] running on GCE node
I20260929 18:44:49.972107 21637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:49.972698 21637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:49.973852 21637 hybrid_clock.cc:648] HybridClock initialized: now 1790707489973831 us; error 31 us; skew 500 ppm
Sep 29 18:44:49 dist-test-slave-smpn krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707489, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:49.976447 21637 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:44:49.977241 21637 webserver.cc:460] Webserver started at http://127.7.157.65:45235/ using document root <none> and password file <none>
I20260929 18:44:49.977747 21637 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:49.977809 21637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:49.977981 21637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:44:49.979955 21637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "c7a318f8ef3449feac4b8e87cf155dce"
format_stamp: "Formatted at 2026-09-29 18:44:49 on dist-test-slave-smpn"
server_key: "48dd3af6e9e67c43dbd6d85550e53db7"
server_key_iv: "fe5361e634622b5ae31d6b86c5afe831"
server_key_version: "encryptionkey@0"
I20260929 18:44:49.980471 21637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "c7a318f8ef3449feac4b8e87cf155dce"
format_stamp: "Formatted at 2026-09-29 18:44:49 on dist-test-slave-smpn"
server_key: "48dd3af6e9e67c43dbd6d85550e53db7"
server_key_iv: "fe5361e634622b5ae31d6b86c5afe831"
server_key_version: "encryptionkey@0"
I20260929 18:44:49.984037 21637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:44:49.986351 21652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:49.987598 21637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:44:49.987718 21637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "c7a318f8ef3449feac4b8e87cf155dce"
format_stamp: "Formatted at 2026-09-29 18:44:49 on dist-test-slave-smpn"
server_key: "48dd3af6e9e67c43dbd6d85550e53db7"
server_key_iv: "fe5361e634622b5ae31d6b86c5afe831"
server_key_version: "encryptionkey@0"
I20260929 18:44:49.987815 21637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:50.000834 21637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:50.004058 21637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:50.004230 21637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:50.004798 21637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:44:50.005616 21637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:44:50.005664 21637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:50.005717 21637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:44:50.005743 21637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:50.013859 21637 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:41593
I20260929 18:44:50.013882 21765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:41593 every 8 connection(s)
I20260929 18:44:50.014696 21637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:44:50.016769 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 21637
I20260929 18:44:50.016857 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:44:50.017107 7797 external_mini_cluster.cc:1428] Setting key 62f710dcc3cc5669f1fcf27f7acf179d
Sep 29 18:44:50 dist-test-slave-smpn krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707489, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
I20260929 18:44:50.028621 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.016407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:48311 (local address 127.7.157.126:32875)
0929 18:44:50.016548 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:50.016552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:50.017385 (+ 833us) server_negotiation.cc:407] Connection header received
0929 18:44:50.018768 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:50.018772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:50.018821 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:50.018906 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:50.020166 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.020692 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.022243 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.022471 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.022553 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.025225 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:50.025228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:50.025230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:50.025252 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:50.026565 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:50.027145 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:50.027148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:50.027149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:50.027228 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:50.027588 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:50.027592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:50.027594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:50.027788 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:50.027875 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:50.028393 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 18:44:50.028494 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:44:50.029424 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.016759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:48311)
0929 18:44:50.017224 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:50.017276 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:50.018566 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:50.019097 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:50.019106 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:50.019451 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:50.020023 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.020042 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.020809 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.020813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.022133 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.022145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.022623 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.022627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.022820 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.023408 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:50.023429 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:50.025064 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:50.026682 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:50.026688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:50.026703 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:50.027028 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:50.027335 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:50.027338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:50.027339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:50.027495 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:50.027909 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:50.027916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:50.028160 (+ 244us) client_negotiation.cc:769] Sending connection context
0929 18:44:50.028328 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 18:44:50.028601 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
I20260929 18:44:50.030587 21766 heartbeater.cc:347] Connected to a master server at 127.7.157.126:32875
I20260929 18:44:50.030835 21766 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:50.031399 21766 heartbeater.cc:511] Master 127.7.157.126:32875 requested a full tablet report, sending...
I20260929 18:44:50.032797 21575 ts_manager.cc:194] Registered new tserver with Master: c7a318f8ef3449feac4b8e87cf155dce (127.7.157.65:41593)
I20260929 18:44:50.033866 21575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:48311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:44:50.044067 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:32875
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:44:50.151782 21771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:50.152056 21771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:50.152099 21771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:50.156301 21771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:50.156371 21771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:50.156430 21771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:44:50.161773 21771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:50.162789 21771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:50.163771 21771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:50.173065 21779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:50.173065 21777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:50.173070 21776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:50.173269 21771 server_base.cc:1034] running on GCE node
I20260929 18:44:50.173699 21771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:50.174317 21771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:50.175472 21771 hybrid_clock.cc:648] HybridClock initialized: now 1790707490175447 us; error 35 us; skew 500 ppm
Sep 29 18:44:50 dist-test-slave-smpn krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707490, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:50.178112 21771 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:44:50.178926 21771 webserver.cc:460] Webserver started at http://127.7.157.66:38445/ using document root <none> and password file <none>
I20260929 18:44:50.179512 21771 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:50.179579 21771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:50.179765 21771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:44:50.181746 21771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "e82a933a7ee14b64923c412b6372d817"
format_stamp: "Formatted at 2026-09-29 18:44:50 on dist-test-slave-smpn"
server_key: "3416ea4626ccbb6712ca84f113433539"
server_key_iv: "9a42edade54ef662c0556c40b032b7ba"
server_key_version: "encryptionkey@0"
I20260929 18:44:50.182274 21771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "e82a933a7ee14b64923c412b6372d817"
format_stamp: "Formatted at 2026-09-29 18:44:50 on dist-test-slave-smpn"
server_key: "3416ea4626ccbb6712ca84f113433539"
server_key_iv: "9a42edade54ef662c0556c40b032b7ba"
server_key_version: "encryptionkey@0"
I20260929 18:44:50.185877 21771 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 18:44:50.188234 21786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:50.189452 21771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:44:50.189554 21771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "e82a933a7ee14b64923c412b6372d817"
format_stamp: "Formatted at 2026-09-29 18:44:50 on dist-test-slave-smpn"
server_key: "3416ea4626ccbb6712ca84f113433539"
server_key_iv: "9a42edade54ef662c0556c40b032b7ba"
server_key_version: "encryptionkey@0"
I20260929 18:44:50.189651 21771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:50.207469 21771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:50.210551 21771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:50.210728 21771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:50.211341 21771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:44:50.212143 21771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:44:50.212189 21771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:50.212242 21771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:44:50.212271 21771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:50.220239 21771 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:39663
I20260929 18:44:50.220287 21899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:39663 every 8 connection(s)
I20260929 18:44:50.221150 21771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:44:50.230134 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 21771
I20260929 18:44:50.230273 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
Sep 29 18:44:50 dist-test-slave-smpn krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707490, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:50.230590 7797 external_mini_cluster.cc:1428] Setting key 1e3cc06c0ce6914d38e0aedb39691f13
I20260929 18:44:50.234645 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.222933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:32965 (local address 127.7.157.126:32875)
0929 18:44:50.223061 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:50.223094 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:50.223974 (+ 880us) server_negotiation.cc:407] Connection header received
0929 18:44:50.225196 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:50.225199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:50.225250 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:50.225332 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:50.226740 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.227376 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.228463 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.228682 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.228750 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.231239 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:50.231243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:50.231246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:50.231273 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:50.232648 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:50.233178 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:50.233181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:50.233182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:50.233252 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:50.233597 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:50.233600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:50.233602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:50.233813 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:50.233902 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:50.234364 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 18:44:50.234490 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:44:50.235440 21902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.223362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:32965)
0929 18:44:50.223813 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:50.223856 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:50.224984 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:50.225533 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:50.225542 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:50.226042 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:50.226594 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.226609 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.227495 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.227498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.228374 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.228383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.228788 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.228791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.228931 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.229453 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:50.229470 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:50.231011 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:50.232760 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:50.232765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:50.232782 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:50.233094 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:50.233321 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:50.233324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:50.233326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:50.233504 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:50.233909 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:50.233914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:50.234149 (+ 235us) client_negotiation.cc:769] Sending connection context
0929 18:44:50.234302 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 18:44:50.234544 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":125,"threads_started":1}
I20260929 18:44:50.236531 21900 heartbeater.cc:347] Connected to a master server at 127.7.157.126:32875
I20260929 18:44:50.236747 21900 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:50.237201 21900 heartbeater.cc:511] Master 127.7.157.126:32875 requested a full tablet report, sending...
I20260929 18:44:50.238193 21575 ts_manager.cc:194] Registered new tserver with Master: e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663)
I20260929 18:44:50.238932 21575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:32965
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:44:50.255602 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:32875
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:44:50.366302 21905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:50.366582 21905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:50.366644 21905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:50.370771 21905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:50.370836 21905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:50.370920 21905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:44:50.376235 21905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:50.377242 21905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:50.378227 21905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:50.387519 21913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:50.387485 21911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:50.387516 21910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:50.387593 21905 server_base.cc:1034] running on GCE node
I20260929 18:44:50.388125 21905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:50.388782 21905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:50.389935 21905 hybrid_clock.cc:648] HybridClock initialized: now 1790707490389915 us; error 34 us; skew 500 ppm
Sep 29 18:44:50 dist-test-slave-smpn krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707490, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:50.392964 21905 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:44:50.393872 21905 webserver.cc:460] Webserver started at http://127.7.157.67:39301/ using document root <none> and password file <none>
I20260929 18:44:50.394448 21905 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:50.394515 21905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:50.394690 21905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:44:50.396855 21905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "7475bf8492804c1593b1bda97b68e5be"
format_stamp: "Formatted at 2026-09-29 18:44:50 on dist-test-slave-smpn"
server_key: "94b570f00835792d3917732ed05c7e62"
server_key_iv: "c3f7671ce7eb860e2182aad9de0a8114"
server_key_version: "encryptionkey@0"
I20260929 18:44:50.397471 21905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "7475bf8492804c1593b1bda97b68e5be"
format_stamp: "Formatted at 2026-09-29 18:44:50 on dist-test-slave-smpn"
server_key: "94b570f00835792d3917732ed05c7e62"
server_key_iv: "c3f7671ce7eb860e2182aad9de0a8114"
server_key_version: "encryptionkey@0"
I20260929 18:44:50.401157 21905 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260929 18:44:50.403553 21920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:50.404711 21905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:44:50.404803 21905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "7475bf8492804c1593b1bda97b68e5be"
format_stamp: "Formatted at 2026-09-29 18:44:50 on dist-test-slave-smpn"
server_key: "94b570f00835792d3917732ed05c7e62"
server_key_iv: "c3f7671ce7eb860e2182aad9de0a8114"
server_key_version: "encryptionkey@0"
I20260929 18:44:50.404886 21905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:50.417268 21905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:50.420003 21905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:50.420164 21905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:50.420701 21905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:44:50.421499 21905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:44:50.421545 21905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:50.421591 21905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:44:50.421629 21905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:50.429662 21905 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:33791
I20260929 18:44:50.429730 22033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:33791 every 8 connection(s)
I20260929 18:44:50.430524 21905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:44:50.431053 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 21905
I20260929 18:44:50.431207 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:44:50.431463 7797 external_mini_cluster.cc:1428] Setting key be9f5ada221f5307133d5904fa765448
Sep 29 18:44:50 dist-test-slave-smpn krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707490, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:50.443267 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.432255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:37789 (local address 127.7.157.126:32875)
0929 18:44:50.432381 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:50.432385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:50.433338 (+ 953us) server_negotiation.cc:407] Connection header received
0929 18:44:50.434681 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:50.434684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:50.434739 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:50.434807 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:50.436056 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.436594 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.437725 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.437949 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.438033 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.440424 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:50.440427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:50.440429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:50.440455 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:50.441540 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:50.442012 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:50.442014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:50.442015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:50.442096 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:50.442344 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:50.442346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:50.442347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:50.442505 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:50.442577 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:50.443003 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 18:44:50.443146 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:44:50.444047 22036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.432756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:37789)
0929 18:44:50.433204 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:50.433247 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:50.434473 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:50.434985 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:50.434992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:50.435418 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:50.435918 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.435932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.436716 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.436720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.437613 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.437622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.438085 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.438089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.438264 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.438774 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:50.438789 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:50.440268 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:50.441662 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:50.441666 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:50.441678 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:50.441935 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:50.442159 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:50.442161 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:50.442162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:50.442282 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:50.442574 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:50.442579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:50.442795 (+ 216us) client_negotiation.cc:769] Sending connection context
0929 18:44:50.442978 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 18:44:50.443237 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":136,"threads_started":1}
I20260929 18:44:50.445138 22034 heartbeater.cc:347] Connected to a master server at 127.7.157.126:32875
I20260929 18:44:50.445371 22034 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:50.445801 22034 heartbeater.cc:511] Master 127.7.157.126:32875 requested a full tablet report, sending...
I20260929 18:44:50.446674 21575 ts_manager.cc:194] Registered new tserver with Master: 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791)
I20260929 18:44:50.447253 21575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:37789
I20260929 18:44:50.457031 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:44:50.465863 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.458904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46088 (local address 127.7.157.126:32875)
0929 18:44:50.459012 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:50.459017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:50.459236 (+ 219us) server_negotiation.cc:407] Connection header received
0929 18:44:50.459355 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:50.459358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:50.459403 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:50.459462 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:50.460129 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.460639 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.461439 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.461675 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.461761 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.463095 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:50.463100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:50.463103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:50.463133 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:50.464251 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:50.464655 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:50.464659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:50.464661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:50.464744 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:50.464990 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:50.464993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:50.464995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:50.465236 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:50.465355 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:50.465586 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 18:44:50.465709 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 18:44:50.468624 21575 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:44:50.471736 21575 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:44:50.486371 22049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.480707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.7.157.67:33791)
0929 18:44:50.481092 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:50.481104 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:50.481330 (+ 226us) server_negotiation.cc:407] Connection header received
0929 18:44:50.481530 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:50.481535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:50.481754 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:50.481878 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:44:50.482282 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.483392 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.484893 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.485542 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.485617 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.485735 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:50.486137 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 18:44:50.486241 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":82,"threads_started":1}
I20260929 18:44:50.486369 22048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.479943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:33791 (local address 127.0.0.1:54612)
0929 18:44:50.481240 (+ 1297us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:50.481253 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:50.481393 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:50.482058 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:50.482061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:50.482078 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:44:50.482165 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.482171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.483548 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.483551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.484765 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.484775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.485816 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.485819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.485937 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.485978 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:44:50.486078 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 18:44:50.486204 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1238,"mutex_wait_us":40,"thread_start_us":90,"threads_started":1}
I20260929 18:44:50.487528 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.479889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:39663 (local address 127.0.0.1:40388)
0929 18:44:50.480322 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:50.480375 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:50.480523 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:50.481389 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:50.481396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:50.481419 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:44:50.481553 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.481559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.483048 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.483052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.485018 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.485027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.486621 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.486624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.487224 (+ 600us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.487251 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 18:44:50.487323 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 18:44:50.487397 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":22,"thread_start_us":80,"threads_started":1}
I20260929 18:44:50.487960 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.480040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.7.157.66:39663)
0929 18:44:50.480514 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:50.480526 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:50.480550 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:44:50.480746 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:50.480752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:50.480990 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:50.481193 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:44:50.481679 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.482900 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.485129 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.486004 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.486104 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.486248 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:50.487715 (+ 1467us) server_negotiation.cc:299] Negotiation successful
0929 18:44:50.487825 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":154,"threads_started":1}
I20260929 18:44:50.488765 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.480648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:41593 (local address 127.0.0.1:51524)
0929 18:44:50.484276 (+ 3628us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:50.484290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:50.484371 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:50.484826 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:50.484829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:50.484845 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:44:50.484955 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.484962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.486272 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.486276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.487265 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.487274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.488431 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.488433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.488504 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.488520 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 18:44:50.488580 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:44:50.488645 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3568,"thread_start_us":98,"threads_started":1}
I20260929 18:44:50.488937 22050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.483491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51524 (local address 127.7.157.65:41593)
0929 18:44:50.484121 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:50.484132 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:50.484467 (+ 335us) server_negotiation.cc:407] Connection header received
0929 18:44:50.484545 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:50.484549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:50.484702 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:50.484831 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:44:50.485064 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.486146 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.487389 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.488170 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.488271 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.488380 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:50.488632 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 18:44:50.488727 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":185,"threads_started":1}
I20260929 18:44:50.489666 21968 tablet_service.cc:1467] Processing CreateTablet for tablet 778ed5c1b6a44724adc0a865bdbfa392 (DEFAULT_TABLE table=test-table [id=3a3030a1967748d2a7de2b3178c788e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:44:50.490887 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778ed5c1b6a44724adc0a865bdbfa392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:44:50.490815 21834 tablet_service.cc:1467] Processing CreateTablet for tablet 778ed5c1b6a44724adc0a865bdbfa392 (DEFAULT_TABLE table=test-table [id=3a3030a1967748d2a7de2b3178c788e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:44:50.491243 21700 tablet_service.cc:1467] Processing CreateTablet for tablet 778ed5c1b6a44724adc0a865bdbfa392 (DEFAULT_TABLE table=test-table [id=3a3030a1967748d2a7de2b3178c788e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:44:50.491919 21834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778ed5c1b6a44724adc0a865bdbfa392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:44:50.492205 21700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778ed5c1b6a44724adc0a865bdbfa392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:44:50.497867 22051 tablet_bootstrap.cc:492] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: Bootstrap starting.
I20260929 18:44:50.498919 22052 tablet_bootstrap.cc:492] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be: Bootstrap starting.
I20260929 18:44:50.499368 22053 tablet_bootstrap.cc:492] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: Bootstrap starting.
I20260929 18:44:50.500051 22051 tablet_bootstrap.cc:654] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: Neither blocks nor log segments found. Creating new log.
I20260929 18:44:50.501032 22051 log.cc:824] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:50.501430 22052 tablet_bootstrap.cc:654] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be: Neither blocks nor log segments found. Creating new log.
I20260929 18:44:50.501713 22053 tablet_bootstrap.cc:654] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: Neither blocks nor log segments found. Creating new log.
I20260929 18:44:50.502355 22052 log.cc:824] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:50.502522 22053 log.cc:824] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:50.503123 22051 tablet_bootstrap.cc:492] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: No bootstrap required, opened a new log
I20260929 18:44:50.503337 22051 ts_tablet_manager.cc:1397] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:44:50.504145 22052 tablet_bootstrap.cc:492] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be: No bootstrap required, opened a new log
I20260929 18:44:50.504321 22052 ts_tablet_manager.cc:1397] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:44:50.505430 22053 tablet_bootstrap.cc:492] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: No bootstrap required, opened a new log
I20260929 18:44:50.505623 22053 ts_tablet_manager.cc:1397] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:44:50.506285 22051 raft_consensus.cc:348] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:50.506512 22051 raft_consensus.cc:374] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:44:50.506583 22051 raft_consensus.cc:729] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a318f8ef3449feac4b8e87cf155dce, State: Initialized, Role: FOLLOWER
I20260929 18:44:50.507001 22051 consensus_queue.cc:260] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:50.507710 21766 heartbeater.cc:503] Master 127.7.157.126:32875 was elected leader, sending a full tablet report...
I20260929 18:44:50.507992 22051 ts_tablet_manager.cc:1428] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:44:50.509337 22053 raft_consensus.cc:348] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:50.509433 22052 raft_consensus.cc:348] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:50.509639 22052 raft_consensus.cc:374] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:44:50.509635 22053 raft_consensus.cc:374] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:44:50.509701 22053 raft_consensus.cc:729] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82a933a7ee14b64923c412b6372d817, State: Initialized, Role: FOLLOWER
I20260929 18:44:50.509735 22052 raft_consensus.cc:729] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7475bf8492804c1593b1bda97b68e5be, State: Initialized, Role: FOLLOWER
I20260929 18:44:50.510174 22052 consensus_queue.cc:260] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:50.510174 22053 consensus_queue.cc:260] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:50.511201 22053 ts_tablet_manager.cc:1428] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:44:50.511462 21900 heartbeater.cc:503] Master 127.7.157.126:32875 was elected leader, sending a full tablet report...
I20260929 18:44:50.511808 22052 ts_tablet_manager.cc:1428] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:44:50.512045 22034 heartbeater.cc:503] Master 127.7.157.126:32875 was elected leader, sending a full tablet report...
W20260929 18:44:50.515934 21767 tablet.cc:2369] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:44:50.532284 22057 raft_consensus.cc:482] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:44:50.532485 22057 raft_consensus.cc:504] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:50.533583 22057 leader_election.cc:288] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663), 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791)
I20260929 18:44:50.537290 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.533916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:60057 (local address 127.7.157.66:39663)
0929 18:44:50.534036 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:50.534040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:50.534284 (+ 244us) server_negotiation.cc:407] Connection header received
0929 18:44:50.534410 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:50.534412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:50.534454 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:50.534512 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:44:50.534770 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.535286 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.536341 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.536730 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.536834 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.536888 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:50.537097 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 18:44:50.537161 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:44:50.537289 22049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.533908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:33525 (local address 127.7.157.67:33791)
0929 18:44:50.534029 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:50.534034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:50.534230 (+ 196us) server_negotiation.cc:407] Connection header received
0929 18:44:50.534352 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:50.534355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:50.534405 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:50.534465 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:44:50.534710 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.535251 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.536351 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.536747 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.536848 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.536898 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:50.537098 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 18:44:50.537169 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:44:50.537312 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.533798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:33791 (local address 127.7.157.65:33525)
0929 18:44:50.534138 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:50.534149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:50.534251 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:50.534483 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:50.534486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:50.534501 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:44:50.534615 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.534620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.535347 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.535350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.536235 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.536244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.536848 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.536851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.536982 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.537010 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 18:44:50.537072 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:44:50.537154 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":44,"thread_start_us":98,"threads_started":1}
I20260929 18:44:50.537319 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.533798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:39663 (local address 127.7.157.65:60057)
0929 18:44:50.534192 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:50.534203 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:50.534303 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:50.534535 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:50.534538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:50.534552 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:44:50.534677 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.534682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.535383 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.535386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.536231 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.536240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.536839 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.536842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.536979 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.537010 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 18:44:50.537081 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 18:44:50.537160 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":103,"thread_start_us":109,"threads_started":1}
I20260929 18:44:50.537873 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778ed5c1b6a44724adc0a865bdbfa392" candidate_uuid: "c7a318f8ef3449feac4b8e87cf155dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7475bf8492804c1593b1bda97b68e5be" is_pre_election: true
I20260929 18:44:50.538146 21988 raft_consensus.cc:2454] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7a318f8ef3449feac4b8e87cf155dce in term 0.
I20260929 18:44:50.538223 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778ed5c1b6a44724adc0a865bdbfa392" candidate_uuid: "c7a318f8ef3449feac4b8e87cf155dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82a933a7ee14b64923c412b6372d817" is_pre_election: true
I20260929 18:44:50.538508 21854 raft_consensus.cc:2454] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7a318f8ef3449feac4b8e87cf155dce in term 0.
I20260929 18:44:50.538564 21655 leader_election.cc:302] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7475bf8492804c1593b1bda97b68e5be, c7a318f8ef3449feac4b8e87cf155dce; no voters:
I20260929 18:44:50.538825 22057 raft_consensus.cc:2781] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:44:50.538897 22057 raft_consensus.cc:482] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:44:50.538928 22057 raft_consensus.cc:3037] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:44:50.539975 22057 raft_consensus.cc:504] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:50.540304 22057 leader_election.cc:288] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 1 election: Requested vote from peers e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663), 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791)
I20260929 18:44:50.540639 21854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778ed5c1b6a44724adc0a865bdbfa392" candidate_uuid: "c7a318f8ef3449feac4b8e87cf155dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82a933a7ee14b64923c412b6372d817"
I20260929 18:44:50.540656 21988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778ed5c1b6a44724adc0a865bdbfa392" candidate_uuid: "c7a318f8ef3449feac4b8e87cf155dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7475bf8492804c1593b1bda97b68e5be"
I20260929 18:44:50.540750 21854 raft_consensus.cc:3037] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:44:50.540759 21988 raft_consensus.cc:3037] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:44:50.541841 21988 raft_consensus.cc:2454] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7a318f8ef3449feac4b8e87cf155dce in term 1.
I20260929 18:44:50.541863 21854 raft_consensus.cc:2454] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7a318f8ef3449feac4b8e87cf155dce in term 1.
I20260929 18:44:50.542098 21655 leader_election.cc:302] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7475bf8492804c1593b1bda97b68e5be, c7a318f8ef3449feac4b8e87cf155dce; no voters:
I20260929 18:44:50.542281 22057 raft_consensus.cc:2781] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:44:50.542517 22057 raft_consensus.cc:686] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 1 LEADER]: Becoming Leader. State: Replica: c7a318f8ef3449feac4b8e87cf155dce, State: Running, Role: LEADER
I20260929 18:44:50.542825 22057 consensus_queue.cc:237] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:50.546073 21575 catalog_manager.cc:5495] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce reported cstate change: term changed from 0 to 1, leader changed from <none> to c7a318f8ef3449feac4b8e87cf155dce (127.7.157.65). New cstate: current_term: 1 leader_uuid: "c7a318f8ef3449feac4b8e87cf155dce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } health_report { overall_health: HEALTHY } } }
I20260929 18:44:50.565820 22050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.562511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.7.157.65:41593)
0929 18:44:50.562641 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:50.562645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:50.562660 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:44:50.562753 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:50.562755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:50.562808 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:50.562901 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:44:50.563141 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.563638 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.564465 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.564677 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.564761 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:50.565044 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:44:50.565464 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:44:50.565568 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:50.565628 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 18:44:50.565686 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:44:50.571807 21988 raft_consensus.cc:1261] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be [term 1 FOLLOWER]: Refusing update from remote peer c7a318f8ef3449feac4b8e87cf155dce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:44:50.571807 21854 raft_consensus.cc:1261] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817 [term 1 FOLLOWER]: Refusing update from remote peer c7a318f8ef3449feac4b8e87cf155dce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:44:50.572506 22062 consensus_queue.cc:1035] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [LEADER]: Connected to new peer: Peer: permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:44:50.572716 22057 consensus_queue.cc:1035] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:44:50.580397 22067 mvcc.cc:205] Tried to move back new op lower bound from 7334737881374560256 to 7334737881266311168. Current Snapshot: MvccSnapshot[applied={T|T < 7334737881374560256}]
I20260929 18:44:50.581032 22066 mvcc.cc:205] Tried to move back new op lower bound from 7334737881374560256 to 7334737881266311168. Current Snapshot: MvccSnapshot[applied={T|T < 7334737881374560256}]
I20260929 18:44:50.586778 21575 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:44:50.586985 21575 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:44:50.589123 21575 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16: Sending DeleteTablet for 3 replicas of tablet 778ed5c1b6a44724adc0a865bdbfa392
I20260929 18:44:50.589854 21700 tablet_service.cc:1514] Processing DeleteTablet for tablet 778ed5c1b6a44724adc0a865bdbfa392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:44:50 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:51524
I20260929 18:44:50.589919 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet 778ed5c1b6a44724adc0a865bdbfa392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:44:50 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:54612
I20260929 18:44:50.590298 22074 tablet_replica.cc:330] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: stopping tablet replica
I20260929 18:44:50.590422 22075 tablet_replica.cc:330] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be: stopping tablet replica
I20260929 18:44:50.590402 21834 tablet_service.cc:1514] Processing DeleteTablet for tablet 778ed5c1b6a44724adc0a865bdbfa392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:44:50 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:40388
I20260929 18:44:50.590634 22074 raft_consensus.cc:2229] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:44:50.590683 22075 raft_consensus.cc:2229] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:44:50.590953 22074 raft_consensus.cc:2258] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:44:50.591056 22075 raft_consensus.cc:2258] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:44:50.591284 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 21637
I20260929 18:44:50.592303 22076 tablet_replica.cc:330] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: stopping tablet replica
I20260929 18:44:50.592561 22076 raft_consensus.cc:2229] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:44:50.592845 22076 raft_consensus.cc:2258] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:44:50.593802 22075 ts_tablet_manager.cc:1905] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:44:50.595945 22075 ts_tablet_manager.cc:1918] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:44:50.596019 22075 log.cc:1196] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/778ed5c1b6a44724adc0a865bdbfa392
I20260929 18:44:50.596256 22075 ts_tablet_manager.cc:1939] T 778ed5c1b6a44724adc0a865bdbfa392 P 7475bf8492804c1593b1bda97b68e5be: Deleting consensus metadata
I20260929 18:44:50.597033 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 21771
W20260929 18:44:50.597035 21559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:41593 (error 108)
I20260929 18:44:50.598712 21561 catalog_manager.cc:4841] TS 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791): tablet 778ed5c1b6a44724adc0a865bdbfa392 (table test-table [id=3a3030a1967748d2a7de2b3178c788e0]) successfully deleted
I20260929 18:44:50.599288 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.598995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:41593 (local address 127.0.0.1:51534)
0929 18:44:50.599141 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:50.599207 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:41593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 18:44:50.599437 21559 catalog_manager.cc:4584] TS c7a318f8ef3449feac4b8e87cf155dce (127.7.157.65:41593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 778ed5c1b6a44724adc0a865bdbfa392: Network error: Client connection negotiation failed: client connection to 127.7.157.65:41593: connect: Connection refused (error 111)
I20260929 18:44:50.601938 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 21905
I20260929 18:44:50.602701 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.602103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:39663 (local address 127.0.0.1:40396)
0929 18:44:50.602576 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:50.602627 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.66:39663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397}
I20260929 18:44:50.606685 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 21543
I20260929 18:44:50.611737 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:32875
--webserver_interface=127.7.157.126
--webserver_port=43827
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:32875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:44:50.719118 22078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:50.719401 22078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:50.719460 22078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:50.723515 22078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:44:50.723587 22078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:50.723614 22078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:50.723635 22078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:44:50.723657 22078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:44:50.729036 22078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:32875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:32875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=43827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:50.730093 22078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:50.731161 22078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:50.743920 22087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:50.743899 22084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:50.744136 22078 server_base.cc:1034] running on GCE node
W20260929 18:44:50.743908 22085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:50.744870 22078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:50.746059 22078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:50.747215 22078 hybrid_clock.cc:648] HybridClock initialized: now 1790707490747181 us; error 49 us; skew 500 ppm
Sep 29 18:44:50 dist-test-slave-smpn krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707490, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:50.749989 22078 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:44:50.750939 22078 webserver.cc:460] Webserver started at http://127.7.157.126:43827/ using document root <none> and password file <none>
I20260929 18:44:50.751539 22078 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:50.751624 22078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:50.755453 22078 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 18:44:50.757560 22094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:50.758811 22078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:44:50.758926 22078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "cab739d0beb54c649c66754a35c9af16"
format_stamp: "Formatted at 2026-09-29 18:44:49 on dist-test-slave-smpn"
server_key: "dbc8063d9d42a95f3968773d7daf4a22"
server_key_iv: "d2ac2cf968083a6e9a966d6eb96f7c83"
server_key_version: "encryptionkey@0"
I20260929 18:44:50.759558 22078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:50.769325 22078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:50.772049 22078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:50.772219 22078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:50.778764 22078 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:32875
I20260929 18:44:50.779044 22146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:32875 every 8 connection(s)
I20260929 18:44:50.779909 22078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:44:50.784293 22147 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:44:50.785943 22147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16: Bootstrap starting.
I20260929 18:44:50.787648 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 22078
I20260929 18:44:50.788107 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:41593
--local_ip_for_outbound_sockets=127.7.157.65
--tserver_master_addrs=127.7.157.126:32875
--webserver_port=45235
--webserver_interface=127.7.157.65
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:44:50.802968 22147 log.cc:824] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:50.811431 22147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:44:50.811761 22147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16: Bootstrap complete.
I20260929 18:44:50.815565 22147 raft_consensus.cc:348] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } }
I20260929 18:44:50.815938 22147 raft_consensus.cc:729] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab739d0beb54c649c66754a35c9af16, State: Initialized, Role: FOLLOWER
I20260929 18:44:50.816406 22147 consensus_queue.cc:260] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } }
I20260929 18:44:50.816519 22147 raft_consensus.cc:388] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:44:50.816581 22147 raft_consensus.cc:482] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:44:50.816672 22147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:44:50.818432 22147 raft_consensus.cc:504] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } }
I20260929 18:44:50.818724 22147 leader_election.cc:302] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cab739d0beb54c649c66754a35c9af16; no voters:
I20260929 18:44:50.818991 22147 leader_election.cc:288] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:44:50.819121 22151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:44:50.819319 22151 raft_consensus.cc:686] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 2 LEADER]: Becoming Leader. State: Replica: cab739d0beb54c649c66754a35c9af16, State: Running, Role: LEADER
I20260929 18:44:50.819654 22151 consensus_queue.cc:237] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } }
I20260929 18:44:50.820190 22147 sys_catalog.cc:564] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:44:50.821026 22153 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cab739d0beb54c649c66754a35c9af16. Latest consensus state: current_term: 2 leader_uuid: "cab739d0beb54c649c66754a35c9af16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } } }
I20260929 18:44:50.821147 22153 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [sys.catalog]: This master's current role is: LEADER
I20260929 18:44:50.821478 22160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:44:50.821503 22152 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cab739d0beb54c649c66754a35c9af16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } } }
I20260929 18:44:50.821622 22152 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [sys.catalog]: This master's current role is: LEADER
I20260929 18:44:50.826185 22160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:44:50.826779 22160 catalog_manager.cc:1259] Loaded cluster ID: 9314cd147c3f404b895f695e4e02ccab
I20260929 18:44:50.826838 22160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:44:50.828054 22160 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 18:44:50.830834 22160 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 18:44:50.834892 22160 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:44:50.835577 22160 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16: Loaded TSK: 0
I20260929 18:44:50.836582 22160 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:44:50.914495 22149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:50.914781 22149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:50.914829 22149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:50.918974 22149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:50.919059 22149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:50.919296 22149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:44:50.925240 22149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:41593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=45235
--tserver_master_addrs=127.7.157.126:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:50.926297 22149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:50.927354 22149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:44:50.938052 22149 server_base.cc:1034] running on GCE node
W20260929 18:44:50.938095 22174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:50.938237 22177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:50.938305 22175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:50.938731 22149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:50.939409 22149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:50.940591 22149 hybrid_clock.cc:648] HybridClock initialized: now 1790707490940560 us; error 49 us; skew 500 ppm
Sep 29 18:44:50 dist-test-slave-smpn krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707490, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:50.943259 22149 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:44:50.944216 22149 webserver.cc:460] Webserver started at http://127.7.157.65:45235/ using document root <none> and password file <none>
I20260929 18:44:50.944742 22149 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:50.944819 22149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:50.948563 22149 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 18:44:50.950685 22184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:50.952004 22149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:44:50.952108 22149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "c7a318f8ef3449feac4b8e87cf155dce"
format_stamp: "Formatted at 2026-09-29 18:44:49 on dist-test-slave-smpn"
server_key: "48dd3af6e9e67c43dbd6d85550e53db7"
server_key_iv: "fe5361e634622b5ae31d6b86c5afe831"
server_key_version: "encryptionkey@0"
I20260929 18:44:50.952653 22149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:50.969579 22149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:50.972743 22149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:50.972923 22149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:50.973548 22149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:44:50.974634 22191 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:44:50.978909 22149 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:44:50.978976 22149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260929 18:44:50.979036 22149 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:44:50.980746 22149 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:44:50.980795 22149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 18:44:50.981058 22191 tablet_bootstrap.cc:492] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: Bootstrap starting.
I20260929 18:44:50.989813 22149 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:41593
I20260929 18:44:50.989890 22298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:41593 every 8 connection(s)
I20260929 18:44:50.990754 22149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:44:50.994755 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 22149
I20260929 18:44:50.995234 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:39663
--local_ip_for_outbound_sockets=127.7.157.66
--tserver_master_addrs=127.7.157.126:32875
--webserver_port=38445
--webserver_interface=127.7.157.66
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:44:51.000540 22191 log.cc:824] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:44:50 dist-test-slave-smpn krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707490, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:51.008251 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.992875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:59137)
0929 18:44:50.993420 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:50.993478 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:50.994585 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:50.996041 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:50.996051 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:50.996529 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:50.997286 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.997308 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.998664 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:50.998668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:50.999970 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:50.999982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.001018 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.001022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:51.001221 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.001848 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:51.001864 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:51.003334 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:51.005273 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:51.005279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:51.005293 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:51.005554 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:51.005881 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:51.005884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:51.005887 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:51.006018 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:51.006562 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:51.006568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:51.006804 (+ 236us) client_negotiation.cc:769] Sending connection context
0929 18:44:51.006964 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 18:44:51.007332 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":138,"threads_started":1}
I20260929 18:44:51.008239 22302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:50.993003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:59137 (local address 127.7.157.126:32875)
0929 18:44:50.993845 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:50.993859 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:50.993897 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:44:50.995098 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:50.995115 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:50.995560 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:50.995925 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:50.997485 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:50.998496 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.000111 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.000878 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.001051 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.003525 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:51.003530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:51.003542 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:51.003581 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:51.005157 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:51.005664 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:51.005669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:51.005673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:51.005793 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:51.006104 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:51.006107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:51.006109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:51.006452 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:51.006580 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:51.007129 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 18:44:51.007395 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":98,"threads_started":1}
I20260929 18:44:51.009507 22191 tablet_bootstrap.cc:492] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:44:51.009945 22191 tablet_bootstrap.cc:492] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: Bootstrap complete.
I20260929 18:44:51.010825 22299 heartbeater.cc:347] Connected to a master server at 127.7.157.126:32875
I20260929 18:44:51.011056 22299 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:51.011317 22191 ts_tablet_manager.cc:1397] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.008s
I20260929 18:44:51.011615 22299 heartbeater.cc:511] Master 127.7.157.126:32875 requested a full tablet report, sending...
I20260929 18:44:51.013909 22111 ts_manager.cc:194] Registered new tserver with Master: c7a318f8ef3449feac4b8e87cf155dce (127.7.157.65:41593)
I20260929 18:44:51.014256 22191 raft_consensus.cc:348] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:51.014796 22191 raft_consensus.cc:729] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a318f8ef3449feac4b8e87cf155dce, State: Initialized, Role: FOLLOWER
I20260929 18:44:51.015367 22191 consensus_queue.cc:260] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:51.016386 22191 ts_tablet_manager.cc:1428] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260929 18:44:51.018003 22111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:59137
I20260929 18:44:51.020673 22299 heartbeater.cc:503] Master 127.7.157.126:32875 was elected leader, sending a full tablet report...
Sep 29 18:44:51 dist-test-slave-smpn krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707490, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for kudu/127.7.157.65@KRBTEST.COM
I20260929 18:44:51.028081 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.016753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:41593 (local address 127.0.0.1:51544)
0929 18:44:51.017141 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:51.017190 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:51.017314 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:51.019222 (+ 1908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:51.019228 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:51.019574 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:51.019681 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:51.019687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.020681 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.020684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:51.021570 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:51.021629 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.022454 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.022457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:51.022537 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.023046 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:51.023059 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:51.024660 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:51.026362 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:51.026371 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:51.026377 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:51.026603 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:51.026877 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:51.026880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:51.026882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:51.026962 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:51.027440 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:51.027446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:51.027566 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 18:44:51.027686 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 18:44:51.027887 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":104,"threads_started":1}
I20260929 18:44:51.028071 22307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.016862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.7.157.65:41593)
0929 18:44:51.018768 (+ 1906us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:51.018780 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:51.018800 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:44:51.018858 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:51.018862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:51.019045 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:51.019197 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:51.019799 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.020584 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.021751 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.022251 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.022344 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.024794 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:51.024803 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:51.024812 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:51.024833 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:51.026257 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:51.026708 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:51.026713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:51.026714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:51.026793 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:51.027039 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:51.027042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:51.027044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:51.027331 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:51.027445 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:51.027775 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 18:44:51.027927 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1810,"thread_start_us":68,"threads_started":1}
I20260929 18:44:51.029129 22233 tablet_service.cc:1514] Processing DeleteTablet for tablet 778ed5c1b6a44724adc0a865bdbfa392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:44:50 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:51544
I20260929 18:44:51.030138 22308 tablet_replica.cc:330] stopping tablet replica
I20260929 18:44:51.030432 22308 raft_consensus.cc:2229] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:44:51.030652 22308 raft_consensus.cc:2258] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:44:51.031610 22308 ts_tablet_manager.cc:1905] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:44:51.035102 22308 ts_tablet_manager.cc:1918] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:44:51.035203 22308 log.cc:1196] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal/wals/778ed5c1b6a44724adc0a865bdbfa392
I20260929 18:44:51.035424 22308 ts_tablet_manager.cc:1939] T 778ed5c1b6a44724adc0a865bdbfa392 P c7a318f8ef3449feac4b8e87cf155dce: Deleting consensus metadata
I20260929 18:44:51.037163 22095 catalog_manager.cc:4841] TS c7a318f8ef3449feac4b8e87cf155dce (127.7.157.65:41593): tablet 778ed5c1b6a44724adc0a865bdbfa392 (table test-table [id=3a3030a1967748d2a7de2b3178c788e0]) successfully deleted
W20260929 18:44:51.112385 22303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:51.112648 22303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:51.112694 22303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:51.116743 22303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:51.116816 22303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:51.116891 22303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:44:51.122403 22303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:39663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=38445
--tserver_master_addrs=127.7.157.126:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:51.123555 22303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:51.124594 22303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:51.134478 22316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:51.134478 22313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:51.134495 22314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:51.134518 22303 server_base.cc:1034] running on GCE node
I20260929 18:44:51.135192 22303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:51.135862 22303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:51.137017 22303 hybrid_clock.cc:648] HybridClock initialized: now 1790707491136996 us; error 34 us; skew 500 ppm
Sep 29 18:44:51 dist-test-slave-smpn krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707491, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:51.139657 22303 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:44:51.140573 22303 webserver.cc:460] Webserver started at http://127.7.157.66:38445/ using document root <none> and password file <none>
I20260929 18:44:51.141124 22303 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:51.141203 22303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:51.145157 22303 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:44:51.147161 22323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:51.148394 22303 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:44:51.148507 22303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "e82a933a7ee14b64923c412b6372d817"
format_stamp: "Formatted at 2026-09-29 18:44:50 on dist-test-slave-smpn"
server_key: "3416ea4626ccbb6712ca84f113433539"
server_key_iv: "9a42edade54ef662c0556c40b032b7ba"
server_key_version: "encryptionkey@0"
I20260929 18:44:51.149019 22303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:51.159346 22303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:51.161923 22303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:51.162086 22303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:51.162689 22303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:44:51.163810 22330 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:44:51.167846 22303 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:44:51.167910 22303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 18:44:51.167973 22303 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:44:51.169575 22303 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:44:51.169618 22303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 18:44:51.169879 22330 tablet_bootstrap.cc:492] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: Bootstrap starting.
I20260929 18:44:51.179096 22303 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:39663
I20260929 18:44:51.179138 22437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:39663 every 8 connection(s)
I20260929 18:44:51.180054 22303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:44:51.182775 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 22303
I20260929 18:44:51.183194 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:33791
--local_ip_for_outbound_sockets=127.7.157.67
--tserver_master_addrs=127.7.157.126:32875
--webserver_port=39301
--webserver_interface=127.7.157.67
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:44:51.187644 22330 log.cc:824] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:44:51 dist-test-slave-smpn krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707491, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:51.195472 22302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.181864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:53587 (local address 127.7.157.126:32875)
0929 18:44:51.182026 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:51.182032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:51.182747 (+ 715us) server_negotiation.cc:407] Connection header received
0929 18:44:51.184105 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:51.184109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:51.184166 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:51.184244 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:51.185908 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.186446 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.188084 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.188450 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.188563 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.191450 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:51.191454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:51.191457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:51.191491 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:51.193149 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:51.193774 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:51.193778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:51.193780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:51.193879 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:51.194255 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:51.194259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:51.194260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:51.194494 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:51.194591 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:51.195157 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 18:44:51.195303 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:44:51.196343 22330 tablet_bootstrap.cc:492] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:44:51.196367 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.181985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:53587)
0929 18:44:51.182545 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:51.182632 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:51.183795 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:51.184439 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:51.184449 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:51.184943 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:51.185742 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:51.185763 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.186597 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.186601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:51.187935 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:51.187946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.188610 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.188613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:51.188762 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.189378 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:51.189399 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:51.191268 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:51.193274 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:51.193280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:51.193294 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:51.193655 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:51.193977 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:51.193980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:51.193982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:51.194147 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:51.194603 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:51.194609 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:51.194884 (+ 275us) client_negotiation.cc:769] Sending connection context
0929 18:44:51.195093 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 18:44:51.195416 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":116,"threads_started":1}
I20260929 18:44:51.196781 22330 tablet_bootstrap.cc:492] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: Bootstrap complete.
I20260929 18:44:51.197582 22438 heartbeater.cc:347] Connected to a master server at 127.7.157.126:32875
I20260929 18:44:51.197827 22438 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:51.197880 22330 ts_tablet_manager.cc:1397] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.008s
I20260929 18:44:51.198380 22438 heartbeater.cc:511] Master 127.7.157.126:32875 requested a full tablet report, sending...
I20260929 18:44:51.200613 22111 ts_manager.cc:194] Registered new tserver with Master: e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663)
I20260929 18:44:51.200884 22330 raft_consensus.cc:348] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:51.201298 22330 raft_consensus.cc:729] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82a933a7ee14b64923c412b6372d817, State: Initialized, Role: FOLLOWER
I20260929 18:44:51.201794 22330 consensus_queue.cc:260] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:51.201910 22111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:53587
I20260929 18:44:51.202790 22330 ts_tablet_manager.cc:1428] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:44:51.204962 22438 heartbeater.cc:503] Master 127.7.157.126:32875 was elected leader, sending a full tablet report...
Sep 29 18:44:51 dist-test-slave-smpn krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707490, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for kudu/127.7.157.66@KRBTEST.COM
I20260929 18:44:51.211839 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.202364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:39663 (local address 127.0.0.1:40412)
0929 18:44:51.202655 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:51.202669 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:51.202747 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:51.203596 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:51.203599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:51.203798 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:51.203903 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:51.203910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.205168 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.205171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:51.205982 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:51.205995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.206659 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.206663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:51.206755 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.207229 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:51.207243 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:51.208526 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:51.210400 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:51.210403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:51.210405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:51.210610 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:51.210882 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:51.210884 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:51.210885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:51.210950 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:51.211421 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:51.211424 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:51.211513 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 18:44:51.211592 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:44:51.211715 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20260929 18:44:51.212006 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.202882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.7.157.66:39663)
0929 18:44:51.203186 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:51.203195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:51.203214 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:44:51.203269 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:51.203274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:51.203451 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:51.203596 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:51.204034 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.205022 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.206115 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.206558 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.206639 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.208669 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:51.208679 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:51.208692 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:51.208716 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:51.210301 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:51.210707 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:51.210712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:51.210714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:51.210803 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:51.211019 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:51.211023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:51.211024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:51.211333 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:51.211457 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:51.211681 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 18:44:51.211842 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":109,"threads_started":1}
I20260929 18:44:51.212967 22372 tablet_service.cc:1514] Processing DeleteTablet for tablet 778ed5c1b6a44724adc0a865bdbfa392 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:44:50 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:40412
I20260929 18:44:51.213907 22445 tablet_replica.cc:330] stopping tablet replica
I20260929 18:44:51.214183 22445 raft_consensus.cc:2229] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:44:51.214396 22445 raft_consensus.cc:2258] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:44:51.215387 22445 ts_tablet_manager.cc:1905] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:44:51.218822 22445 ts_tablet_manager.cc:1918] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:44:51.218911 22445 log.cc:1196] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/778ed5c1b6a44724adc0a865bdbfa392
I20260929 18:44:51.219169 22445 ts_tablet_manager.cc:1939] T 778ed5c1b6a44724adc0a865bdbfa392 P e82a933a7ee14b64923c412b6372d817: Deleting consensus metadata
I20260929 18:44:51.220680 22095 catalog_manager.cc:4841] TS e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663): tablet 778ed5c1b6a44724adc0a865bdbfa392 (table test-table [id=3a3030a1967748d2a7de2b3178c788e0]) successfully deleted
W20260929 18:44:51.299423 22441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:51.299772 22441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:51.299820 22441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:51.303920 22441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:51.304006 22441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:51.304087 22441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:44:51.309585 22441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:33791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=39301
--tserver_master_addrs=127.7.157.126:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:51.310719 22441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:51.311827 22441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:51.321444 22451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:51.321658 22450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:51.321507 22453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:51.321751 22441 server_base.cc:1034] running on GCE node
I20260929 18:44:51.322167 22441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:51.322818 22441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:51.323968 22441 hybrid_clock.cc:648] HybridClock initialized: now 1790707491323935 us; error 46 us; skew 500 ppm
Sep 29 18:44:51 dist-test-slave-smpn krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707491, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:51.326566 22441 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:44:51.327505 22441 webserver.cc:460] Webserver started at http://127.7.157.67:39301/ using document root <none> and password file <none>
I20260929 18:44:51.328038 22441 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:51.328120 22441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:51.332015 22441 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 18:44:51.334120 22460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:51.335369 22441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:44:51.335525 22441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "7475bf8492804c1593b1bda97b68e5be"
format_stamp: "Formatted at 2026-09-29 18:44:50 on dist-test-slave-smpn"
server_key: "94b570f00835792d3917732ed05c7e62"
server_key_iv: "c3f7671ce7eb860e2182aad9de0a8114"
server_key_version: "encryptionkey@0"
I20260929 18:44:51.336054 22441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:51.348472 22441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:51.351377 22441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:51.351538 22441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:51.352128 22441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:44:51.352922 22441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:44:51.352977 22441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:51.353039 22441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:44:51.353061 22441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:51.361553 22441 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:33791
I20260929 18:44:51.361598 22573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:33791 every 8 connection(s)
I20260929 18:44:51.362571 22441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:44:51.370808 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 22441
Sep 29 18:44:51 dist-test-slave-smpn krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707491, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:44:51.375813 22302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.364296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:53069 (local address 127.7.157.126:32875)
0929 18:44:51.364454 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:51.364460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:51.365282 (+ 822us) server_negotiation.cc:407] Connection header received
0929 18:44:51.366478 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:51.366481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:51.366525 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:51.366593 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:51.367728 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.368263 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.369298 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.369510 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.369578 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.372052 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:51.372056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:51.372059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:51.372082 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:51.373653 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:51.374212 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:51.374215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:51.374216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:51.374290 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:51.374630 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:51.374632 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:51.374633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:51.374867 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:51.374958 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:51.375535 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 18:44:51.375653 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":59}
I20260929 18:44:51.376886 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.364717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:53069)
0929 18:44:51.365138 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:51.365179 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:51.366294 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:51.366743 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:51.366750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:51.367115 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:51.367590 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:51.367603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.368350 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.368352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:51.369209 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:51.369217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.369629 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.369631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:51.369763 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.370300 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:51.370323 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:51.371901 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:51.373764 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:51.373770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:51.373788 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:51.374123 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:51.374365 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:51.374368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:51.374370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:51.374541 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:51.374960 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:51.374965 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:51.375264 (+ 299us) client_negotiation.cc:769] Sending connection context
0929 18:44:51.375482 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 18:44:51.375824 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":110,"threads_started":1}
I20260929 18:44:51.378099 22574 heartbeater.cc:347] Connected to a master server at 127.7.157.126:32875
I20260929 18:44:51.378328 22574 heartbeater.cc:464] Registering TS with master...
I20260929 18:44:51.378798 22574 heartbeater.cc:511] Master 127.7.157.126:32875 requested a full tablet report, sending...
I20260929 18:44:51.379865 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.371474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46094 (local address 127.7.157.126:32875)
0929 18:44:51.371710 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:51.371714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:51.371848 (+ 134us) server_negotiation.cc:407] Connection header received
0929 18:44:51.371977 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:51.371980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:51.372026 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:51.372115 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:51.372915 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.373698 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.374734 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.375127 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.375205 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.376641 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:51.376644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:51.376646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:51.376666 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:51.378342 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:51.378741 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:51.378744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:51.378746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:51.378823 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:51.379056 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:51.379059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:51.379060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:51.379250 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:51.379431 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:51.379608 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 18:44:51.379723 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
I20260929 18:44:51.379869 22111 ts_manager.cc:194] Registered new tserver with Master: 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791)
I20260929 18:44:51.380456 22111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:53069
I20260929 18:44:51.381558 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:44:51.385548 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.382149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46100 (local address 127.7.157.126:32875)
0929 18:44:51.382261 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:51.382266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:51.382404 (+ 138us) server_negotiation.cc:407] Connection header received
0929 18:44:51.382507 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:51.382509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:51.382588 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:51.382668 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:44:51.382818 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.383354 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.384123 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.384360 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.384445 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.384764 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:44:51.385212 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:44:51.385315 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:51.385373 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 18:44:51.385428 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:44:51.386101 22111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:44:51.389276 22111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:44:51.396251 22233 tablet_service.cc:1467] Processing CreateTablet for tablet 704c83a269674ec4a7986768eea013fa (DEFAULT_TABLE table=test-table [id=864427655ddd4e1a8e00d5ae503d23de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:44:51.396531 22372 tablet_service.cc:1467] Processing CreateTablet for tablet 704c83a269674ec4a7986768eea013fa (DEFAULT_TABLE table=test-table [id=864427655ddd4e1a8e00d5ae503d23de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:44:51.396926 22233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 704c83a269674ec4a7986768eea013fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:44:51.397033 22372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 704c83a269674ec4a7986768eea013fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:44:51.399669 22191 tablet_bootstrap.cc:492] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce: Bootstrap starting.
I20260929 18:44:51.399756 22330 tablet_bootstrap.cc:492] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817: Bootstrap starting.
I20260929 18:44:51.400420 22191 tablet_bootstrap.cc:654] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce: Neither blocks nor log segments found. Creating new log.
I20260929 18:44:51.400584 22330 tablet_bootstrap.cc:654] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817: Neither blocks nor log segments found. Creating new log.
I20260929 18:44:51.401391 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.395489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:33791 (local address 127.0.0.1:54616)
0929 18:44:51.395589 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:51.395601 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:51.395668 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:51.396790 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:51.396792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:51.396809 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:44:51.396912 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:51.396919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.398478 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.398481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:51.399670 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:51.399682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.400927 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.400931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:51.401032 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.401084 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 18:44:51.401167 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:44:51.401256 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 18:44:51.401460 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.395919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.7.157.67:33791)
0929 18:44:51.396261 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:51.396274 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:51.396297 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:44:51.396375 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:51.396380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:51.396615 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:51.397066 (+ 451us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:44:51.397215 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.398312 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.399882 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.400771 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.400888 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.401063 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:51.401213 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 18:44:51.401315 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":130,"threads_started":1}
I20260929 18:44:51.402057 22330 tablet_bootstrap.cc:492] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817: No bootstrap required, opened a new log
I20260929 18:44:51.402143 22330 ts_tablet_manager.cc:1397] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:44:51.402268 22191 tablet_bootstrap.cc:492] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce: No bootstrap required, opened a new log
I20260929 18:44:51.402360 22191 ts_tablet_manager.cc:1397] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 18:44:51.402591 22330 raft_consensus.cc:348] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:51.402704 22330 raft_consensus.cc:374] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:44:51.402738 22330 raft_consensus.cc:729] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82a933a7ee14b64923c412b6372d817, State: Initialized, Role: FOLLOWER
I20260929 18:44:51.402813 22191 raft_consensus.cc:348] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:51.402856 22330 consensus_queue.cc:260] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:51.402921 22191 raft_consensus.cc:374] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:44:51.402966 22191 raft_consensus.cc:729] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a318f8ef3449feac4b8e87cf155dce, State: Initialized, Role: FOLLOWER
I20260929 18:44:51.403107 22191 consensus_queue.cc:260] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:51.403208 22330 ts_tablet_manager.cc:1428] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:44:51.403348 22191 ts_tablet_manager.cc:1428] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:44:51.404789 22508 tablet_service.cc:1467] Processing CreateTablet for tablet 704c83a269674ec4a7986768eea013fa (DEFAULT_TABLE table=test-table [id=864427655ddd4e1a8e00d5ae503d23de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:44:51.405969 22508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 704c83a269674ec4a7986768eea013fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:44:51.409500 22305 raft_consensus.cc:482] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:44:51.409685 22305 raft_consensus.cc:504] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:51.410719 22305 leader_election.cc:288] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791), e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663)
I20260929 18:44:51.414700 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.410897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:33791 (local address 127.7.157.65:39287)
0929 18:44:51.411096 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:51.411112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:51.411210 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:51.411766 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:51.411770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:51.411788 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:44:51.411894 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:51.411899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.412671 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.412674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:51.413648 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:51.413658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.414244 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.414248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:51.414353 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.414402 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 18:44:51.414477 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:44:51.414552 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 18:44:51.414705 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.411466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:39287 (local address 127.7.157.67:33791)
0929 18:44:51.411577 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:51.411581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:51.411594 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:44:51.411636 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:51.411638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:51.411674 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:51.411740 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:44:51.411991 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.412577 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.413761 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.414143 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.414244 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.414298 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:51.414515 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 18:44:51.414591 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 18:44:51.414731 22587 tablet_bootstrap.cc:492] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be: Bootstrap starting.
I20260929 18:44:51.415405 22528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704c83a269674ec4a7986768eea013fa" candidate_uuid: "c7a318f8ef3449feac4b8e87cf155dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7475bf8492804c1593b1bda97b68e5be" is_pre_election: true
I20260929 18:44:51.417169 22587 tablet_bootstrap.cc:654] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be: Neither blocks nor log segments found. Creating new log.
W20260929 18:44:51.417095 22187 leader_election.cc:341] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791): Illegal state: must be running to vote when last-logged opid is not known
I20260929 18:44:51.417272 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.410959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:39663 (local address 127.7.157.65:47407)
0929 18:44:51.411307 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:51.411319 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:51.411391 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:51.411998 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:51.412001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:51.412019 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:44:51.412136 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:51.412142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.413267 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.413271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:51.414313 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:51.414323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.416734 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.416737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:51.416852 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.416876 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:44:51.416948 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 18:44:51.417023 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":158,"threads_started":1}
I20260929 18:44:51.417356 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.411381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:47407 (local address 127.7.157.66:39663)
0929 18:44:51.411509 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:51.411513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:51.411527 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:44:51.411577 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:51.411580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:51.411641 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:51.411743 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:44:51.412262 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.413012 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.414414 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.415033 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.416517 (+ 1484us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.416673 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:51.417155 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 18:44:51.417243 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:44:51.417882 22392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704c83a269674ec4a7986768eea013fa" candidate_uuid: "c7a318f8ef3449feac4b8e87cf155dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82a933a7ee14b64923c412b6372d817" is_pre_election: true
I20260929 18:44:51.418211 22392 raft_consensus.cc:2454] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7a318f8ef3449feac4b8e87cf155dce in term 0.
I20260929 18:44:51.418370 22587 log.cc:824] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:51.418622 22185 leader_election.cc:302] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c7a318f8ef3449feac4b8e87cf155dce, e82a933a7ee14b64923c412b6372d817; no voters: 7475bf8492804c1593b1bda97b68e5be
I20260929 18:44:51.418919 22305 raft_consensus.cc:2781] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:44:51.419025 22305 raft_consensus.cc:482] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:44:51.419121 22305 raft_consensus.cc:3037] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:44:51.420147 22305 raft_consensus.cc:504] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:51.420401 22587 tablet_bootstrap.cc:492] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be: No bootstrap required, opened a new log
I20260929 18:44:51.420500 22305 leader_election.cc:288] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 1 election: Requested vote from peers 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791), e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663)
I20260929 18:44:51.420636 22587 ts_tablet_manager.cc:1397] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:44:51.420872 22528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704c83a269674ec4a7986768eea013fa" candidate_uuid: "c7a318f8ef3449feac4b8e87cf155dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7475bf8492804c1593b1bda97b68e5be"
I20260929 18:44:51.421016 22392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704c83a269674ec4a7986768eea013fa" candidate_uuid: "c7a318f8ef3449feac4b8e87cf155dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82a933a7ee14b64923c412b6372d817"
I20260929 18:44:51.421159 22392 raft_consensus.cc:3037] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817 [term 0 FOLLOWER]: Advancing to term 1
W20260929 18:44:51.421412 22187 leader_election.cc:341] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791): Illegal state: must be running to vote when last-logged opid is not known
I20260929 18:44:51.422648 22392 raft_consensus.cc:2454] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7a318f8ef3449feac4b8e87cf155dce in term 1.
I20260929 18:44:51.422979 22185 leader_election.cc:302] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c7a318f8ef3449feac4b8e87cf155dce, e82a933a7ee14b64923c412b6372d817; no voters: 7475bf8492804c1593b1bda97b68e5be
I20260929 18:44:51.423240 22305 raft_consensus.cc:2781] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:44:51.423457 22305 raft_consensus.cc:686] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 LEADER]: Becoming Leader. State: Replica: c7a318f8ef3449feac4b8e87cf155dce, State: Running, Role: LEADER
I20260929 18:44:51.423731 22305 consensus_queue.cc:237] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:51.423916 22587 raft_consensus.cc:348] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:51.424140 22587 raft_consensus.cc:374] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:44:51.424186 22587 raft_consensus.cc:729] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7475bf8492804c1593b1bda97b68e5be, State: Initialized, Role: FOLLOWER
I20260929 18:44:51.424641 22587 consensus_queue.cc:260] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:51.425295 22574 heartbeater.cc:503] Master 127.7.157.126:32875 was elected leader, sending a full tablet report...
I20260929 18:44:51.426968 22587 ts_tablet_manager.cc:1428] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:44:51.426717 22109 catalog_manager.cc:5495] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce reported cstate change: term changed from 0 to 1, leader changed from <none> to c7a318f8ef3449feac4b8e87cf155dce (127.7.157.65). New cstate: current_term: 1 leader_uuid: "c7a318f8ef3449feac4b8e87cf155dce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } health_report { overall_health: HEALTHY } } }
W20260929 18:44:51.431005 22439 tablet.cc:2369] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:44:51.441664 22307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.438109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.7.157.65:41593)
0929 18:44:51.438237 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:51.438241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:51.438255 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:44:51.438353 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:51.438356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:51.438411 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:51.438495 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:44:51.438680 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.439378 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.440248 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:51.440567 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:51.440650 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:51.440874 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:44:51.441314 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:44:51.441401 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:51.441451 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 18:44:51.441506 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:44:51.445451 22528 raft_consensus.cc:3037] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:44:51.445619 22392 raft_consensus.cc:1261] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817 [term 1 FOLLOWER]: Refusing update from remote peer c7a318f8ef3449feac4b8e87cf155dce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:44:51.446322 22589 consensus_queue.cc:1035] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [LEADER]: Connected to new peer: Peer: permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:44:51.446818 22528 raft_consensus.cc:1261] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be [term 1 FOLLOWER]: Refusing update from remote peer c7a318f8ef3449feac4b8e87cf155dce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:44:51.447451 22589 consensus_queue.cc:1035] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:44:51.454664 22593 mvcc.cc:205] Tried to move back new op lower bound from 7334737884955025408 to 7334737884874305536. Current Snapshot: MvccSnapshot[applied={T|T < 7334737884955025408}]
I20260929 18:44:51.457892 22595 mvcc.cc:205] Tried to move back new op lower bound from 7334737884955025408 to 7334737884874305536. Current Snapshot: MvccSnapshot[applied={T|T < 7334737884955025408}]
I20260929 18:44:51.459519 22111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:44:51.459683 22111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:44:51.461833 22111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16: Sending DeleteTablet for 3 replicas of tablet 704c83a269674ec4a7986768eea013fa
I20260929 18:44:51.462541 22508 tablet_service.cc:1514] Processing DeleteTablet for tablet 704c83a269674ec4a7986768eea013fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:44:51 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:54616
I20260929 18:44:51.462678 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 22149
I20260929 18:44:51.462743 22372 tablet_service.cc:1514] Processing DeleteTablet for tablet 704c83a269674ec4a7986768eea013fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:44:51 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:40412
I20260929 18:44:51.463325 22445 tablet_replica.cc:330] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817: stopping tablet replica
I20260929 18:44:51.463447 22445 raft_consensus.cc:2229] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:44:51.463578 22445 raft_consensus.cc:2258] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:44:51.464006 22601 tablet_replica.cc:330] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be: stopping tablet replica
I20260929 18:44:51.464308 22601 raft_consensus.cc:2229] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:44:51.464624 22601 raft_consensus.cc:2258] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:44:51.465041 22445 ts_tablet_manager.cc:1905] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:44:51.465987 22601 ts_tablet_manager.cc:1905] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:44:51.467121 22445 ts_tablet_manager.cc:1918] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:44:51.467193 22445 log.cc:1196] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/704c83a269674ec4a7986768eea013fa
I20260929 18:44:51.467407 22445 ts_tablet_manager.cc:1939] T 704c83a269674ec4a7986768eea013fa P e82a933a7ee14b64923c412b6372d817: Deleting consensus metadata
I20260929 18:44:51.468024 22095 catalog_manager.cc:4841] TS e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663): tablet 704c83a269674ec4a7986768eea013fa (table test-table [id=864427655ddd4e1a8e00d5ae503d23de]) successfully deleted
I20260929 18:44:51.468530 22601 ts_tablet_manager.cc:1918] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:44:51.468643 22601 log.cc:1196] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/704c83a269674ec4a7986768eea013fa
I20260929 18:44:51.468956 22601 ts_tablet_manager.cc:1939] T 704c83a269674ec4a7986768eea013fa P 7475bf8492804c1593b1bda97b68e5be: Deleting consensus metadata
W20260929 18:44:51.469825 22095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:41593 (error 108)
I20260929 18:44:51.469924 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 22303
I20260929 18:44:51.470744 22097 catalog_manager.cc:4841] TS 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791): tablet 704c83a269674ec4a7986768eea013fa (table test-table [id=864427655ddd4e1a8e00d5ae503d23de]) successfully deleted
I20260929 18:44:51.471362 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:51.471039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:41593 (local address 127.0.0.1:51568)
0929 18:44:51.471194 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:51.471275 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:41593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 18:44:51.471524 22095 catalog_manager.cc:4584] TS c7a318f8ef3449feac4b8e87cf155dce (127.7.157.65:41593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 704c83a269674ec4a7986768eea013fa: Network error: Client connection negotiation failed: client connection to 127.7.157.65:41593: connect: Connection refused (error 111)
I20260929 18:44:51.475533 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 22441
I20260929 18:44:51.480026 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 22078
I20260929 18:44:51.486088 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:32875
--webserver_interface=127.7.157.126
--webserver_port=43827
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:32875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:44:51.596900 22603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:51.597185 22603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:51.597226 22603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:51.601253 22603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:44:51.601321 22603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:51.601341 22603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:51.601356 22603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:44:51.601382 22603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:44:51.606858 22603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:32875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:32875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=43827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:51.608014 22603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:51.609088 22603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:51.617535 22608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:51.617595 22611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:51.617535 22609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:51.617609 22603 server_base.cc:1034] running on GCE node
I20260929 18:44:51.618407 22603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:51.619635 22603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:51.620807 22603 hybrid_clock.cc:648] HybridClock initialized: now 1790707491620787 us; error 33 us; skew 500 ppm
Sep 29 18:44:51 dist-test-slave-smpn krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707491, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:51.623564 22603 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:44:51.624473 22603 webserver.cc:460] Webserver started at http://127.7.157.126:43827/ using document root <none> and password file <none>
I20260929 18:44:51.625059 22603 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:51.625133 22603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:51.629197 22603 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:44:51.631469 22618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:51.632735 22603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:44:51.632855 22603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "cab739d0beb54c649c66754a35c9af16"
format_stamp: "Formatted at 2026-09-29 18:44:49 on dist-test-slave-smpn"
server_key: "dbc8063d9d42a95f3968773d7daf4a22"
server_key_iv: "d2ac2cf968083a6e9a966d6eb96f7c83"
server_key_version: "encryptionkey@0"
I20260929 18:44:51.633430 22603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:51.657526 22603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:51.660751 22603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:51.660934 22603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:51.667723 22603 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:32875
I20260929 18:44:51.667974 22670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:32875 every 8 connection(s)
I20260929 18:44:51.668860 22603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:44:51.672539 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 22603
I20260929 18:44:51.672941 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:41593
--local_ip_for_outbound_sockets=127.7.157.65
--tserver_master_addrs=127.7.157.126:32875
--webserver_port=45235
--webserver_interface=127.7.157.65
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:44:51.673480 22671 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:44:51.675277 22671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16: Bootstrap starting.
I20260929 18:44:51.692389 22671 log.cc:824] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:51.703323 22671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:44:51.703685 22671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16: Bootstrap complete.
I20260929 18:44:51.707510 22671 raft_consensus.cc:348] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } }
I20260929 18:44:51.707924 22671 raft_consensus.cc:729] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab739d0beb54c649c66754a35c9af16, State: Initialized, Role: FOLLOWER
I20260929 18:44:51.708400 22671 consensus_queue.cc:260] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } }
I20260929 18:44:51.708524 22671 raft_consensus.cc:388] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:44:51.708593 22671 raft_consensus.cc:482] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:44:51.708685 22671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:44:51.710613 22671 raft_consensus.cc:504] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } }
I20260929 18:44:51.710948 22671 leader_election.cc:302] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cab739d0beb54c649c66754a35c9af16; no voters:
I20260929 18:44:51.711295 22671 leader_election.cc:288] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 18:44:51.711413 22675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:44:51.711617 22675 raft_consensus.cc:686] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [term 3 LEADER]: Becoming Leader. State: Replica: cab739d0beb54c649c66754a35c9af16, State: Running, Role: LEADER
I20260929 18:44:51.711920 22675 consensus_queue.cc:237] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } }
I20260929 18:44:51.712564 22671 sys_catalog.cc:564] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:44:51.713268 22676 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cab739d0beb54c649c66754a35c9af16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } } }
I20260929 18:44:51.713399 22676 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [sys.catalog]: This master's current role is: LEADER
I20260929 18:44:51.713678 22677 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cab739d0beb54c649c66754a35c9af16. Latest consensus state: current_term: 3 leader_uuid: "cab739d0beb54c649c66754a35c9af16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab739d0beb54c649c66754a35c9af16" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32875 } } }
I20260929 18:44:51.713759 22677 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab739d0beb54c649c66754a35c9af16 [sys.catalog]: This master's current role is: LEADER
I20260929 18:44:51.714192 22682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:44:51.719477 22682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:44:51.720186 22682 catalog_manager.cc:1259] Loaded cluster ID: 9314cd147c3f404b895f695e4e02ccab
I20260929 18:44:51.720252 22682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 18:44:51.721300 22682 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6ba7cac9ad google::LogMessage::Fail() at ??:0
@ 0x7f6ba7caf88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6ba7cac4de google::LogMessage::Flush() at ??:0
@ 0x7f6ba7cada19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6bb25ebfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6bb25ecd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6bb25e9579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6bb262d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6ba9efe3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6ba9efec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6ba9f003f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6ba9eef512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6bab5336db start_thread at ??:0
@ 0x7f6ba67aa61f clone at ??:0
W20260929 18:44:51.810946 22673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:51.811393 22673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:51.811483 22673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:51.817535 22673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:51.817636 22673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:51.817729 22673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:44:51.825836 22673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:41593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=45235
--tserver_master_addrs=127.7.157.126:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:51.827410 22673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:51.828886 22673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:51.840313 22713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:51.840323 22712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:51.840442 22715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:51.840698 22673 server_base.cc:1034] running on GCE node
I20260929 18:44:51.841212 22673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:51.842077 22673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:51.843267 22673 hybrid_clock.cc:648] HybridClock initialized: now 1790707491843223 us; error 49 us; skew 500 ppm
Sep 29 18:44:51 dist-test-slave-smpn krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707491, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:51.846684 22673 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:44:51.847890 22673 webserver.cc:460] Webserver started at http://127.7.157.65:45235/ using document root <none> and password file <none>
I20260929 18:44:51.848662 22673 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:51.848766 22673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:51.854149 22673 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:44:51.856968 22722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:51.858515 22673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:44:51.858645 22673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "c7a318f8ef3449feac4b8e87cf155dce"
format_stamp: "Formatted at 2026-09-29 18:44:49 on dist-test-slave-smpn"
server_key: "48dd3af6e9e67c43dbd6d85550e53db7"
server_key_iv: "fe5361e634622b5ae31d6b86c5afe831"
server_key_version: "encryptionkey@0"
I20260929 18:44:51.859282 22673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:51.880934 22673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:51.884066 22673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:51.884225 22673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:51.884832 22673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:44:51.885967 22729 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:44:51.890992 22673 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:44:51.891111 22673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 18:44:51.891204 22673 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:44:51.892957 22673 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:44:51.893011 22673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 18:44:51.893335 22729 tablet_bootstrap.cc:492] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce: Bootstrap starting.
I20260929 18:44:51.903752 22673 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:41593
I20260929 18:44:51.904060 22836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:41593 every 8 connection(s)
I20260929 18:44:51.904866 22673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:44:51.909747 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 22673
I20260929 18:44:51.910132 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:39663
--local_ip_for_outbound_sockets=127.7.157.66
--tserver_master_addrs=127.7.157.126:32875
--webserver_port=38445
--webserver_interface=127.7.157.66
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:44:51.917241 22729 log.cc:824] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce: Log is configured to *not* fsync() on all Append() calls
I20260929 18:44:51.924819 22729 tablet_bootstrap.cc:492] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:44:51.925168 22729 tablet_bootstrap.cc:492] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce: Bootstrap complete.
I20260929 18:44:51.926086 22729 ts_tablet_manager.cc:1397] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.004s
I20260929 18:44:51.928567 22729 raft_consensus.cc:348] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:51.928988 22729 raft_consensus.cc:729] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a318f8ef3449feac4b8e87cf155dce, State: Initialized, Role: FOLLOWER
I20260929 18:44:51.929474 22729 consensus_queue.cc:260] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:51.930440 22729 ts_tablet_manager.cc:1428] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260929 18:44:52.021369 22839 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:44:51.906904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:56037)
0929 18:44:51.907474 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:51.907530 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:51.908389 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:52.020581 (+112192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":188,"threads_started":1}
W20260929 18:44:52.021983 22837 heartbeater.cc:644] Failed to heartbeat to 127.7.157.126:32875 (0 consecutive failures): Network error: Failed to ping master at 127.7.157.126:32875: Client connection negotiation failed: client connection to 127.7.157.126:32875: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:44:52.022707 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:52.022455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:33893)
0929 18:44:52.022582 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:52.022647 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 18:44:52.023465 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:52.023260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:48751)
0929 18:44:52.023369 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:52.023412 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 18:44:52.023617 22837 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:44:52.049507 22840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:52.049750 22840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:52.049793 22840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:52.053781 22840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:52.053838 22840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:52.053893 22840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:44:52.059085 22840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:39663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=38445
--tserver_master_addrs=127.7.157.126:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:52.060109 22840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:52.061082 22840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:52.070590 22848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:52.070964 22840 server_base.cc:1034] running on GCE node
W20260929 18:44:52.070698 22847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:52.070593 22850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:52.073375 22840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:52.074074 22840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:52.075219 22840 hybrid_clock.cc:648] HybridClock initialized: now 1790707492075149 us; error 88 us; skew 500 ppm
Sep 29 18:44:52 dist-test-slave-smpn krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707492, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:52.078150 22840 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:44:52.079018 22840 webserver.cc:460] Webserver started at http://127.7.157.66:38445/ using document root <none> and password file <none>
I20260929 18:44:52.079574 22840 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:52.079634 22840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:52.083465 22840 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:44:52.085510 22857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:52.086714 22840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:44:52.086815 22840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "e82a933a7ee14b64923c412b6372d817"
format_stamp: "Formatted at 2026-09-29 18:44:50 on dist-test-slave-smpn"
server_key: "3416ea4626ccbb6712ca84f113433539"
server_key_iv: "9a42edade54ef662c0556c40b032b7ba"
server_key_version: "encryptionkey@0"
I20260929 18:44:52.087420 22840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:52.101189 22840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:52.104106 22840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:52.104254 22840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:52.104771 22840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:44:52.105556 22840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:44:52.105602 22840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:52.105646 22840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:44:52.105679 22840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:52.114368 22840 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:39663
I20260929 18:44:52.114455 22970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:39663 every 8 connection(s)
I20260929 18:44:52.115480 22840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:44:52.119416 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:52.117636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:33701)
0929 18:44:52.118095 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:52.118208 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
W20260929 18:44:52.119894 22971 heartbeater.cc:644] Failed to heartbeat to 127.7.157.126:32875 (0 consecutive failures): Network error: Failed to ping master at 127.7.157.126:32875: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
I20260929 18:44:52.120115 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 22840
I20260929 18:44:52.120527 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:52.120272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:47355)
0929 18:44:52.120387 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:52.120443 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 18:44:52.120518 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:33791
--local_ip_for_outbound_sockets=127.7.157.67
--tserver_master_addrs=127.7.157.126:32875
--webserver_port=39301
--webserver_interface=127.7.157.67
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:44:52.121248 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:52.121044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:57607)
0929 18:44:52.121147 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:52.121193 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 18:44:52.121403 22971 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:44:52.229321 22974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:44:52.229583 22974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:44:52.229624 22974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:44:52.233640 22974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:44:52.233697 22974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:44:52.233757 22974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:44:52.238890 22974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:33791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=39301
--tserver_master_addrs=127.7.157.126:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:44:52.239883 22974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:44:52.240828 22974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:44:52.250131 22980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:44:52.250128 22979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:52.250278 22974 server_base.cc:1034] running on GCE node
W20260929 18:44:52.250137 22982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:44:52.250811 22974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:44:52.251502 22974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:44:52.252702 22974 hybrid_clock.cc:648] HybridClock initialized: now 1790707492252676 us; error 41 us; skew 500 ppm
Sep 29 18:44:52 dist-test-slave-smpn krb5kdc[21530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707492, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:44:52.255577 22974 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:44:52.256532 22974 webserver.cc:460] Webserver started at http://127.7.157.67:39301/ using document root <none> and password file <none>
I20260929 18:44:52.257079 22974 fs_manager.cc:359] Metadata directory not provided
I20260929 18:44:52.257145 22974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:44:52.261206 22974 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:44:52.263345 22989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:52.264654 22974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:44:52.264760 22974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "7475bf8492804c1593b1bda97b68e5be"
format_stamp: "Formatted at 2026-09-29 18:44:50 on dist-test-slave-smpn"
server_key: "94b570f00835792d3917732ed05c7e62"
server_key_iv: "c3f7671ce7eb860e2182aad9de0a8114"
server_key_version: "encryptionkey@0"
I20260929 18:44:52.265285 22974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:44:52.295768 22974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:44:52.299538 22974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:44:52.299743 22974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:44:52.300472 22974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:44:52.301496 22974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:44:52.301556 22974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:52.301632 22974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:44:52.301661 22974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:44:52.310070 22974 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:33791
I20260929 18:44:52.310123 23102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:33791 every 8 connection(s)
I20260929 18:44:52.311096 22974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:44:52.314967 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:52.313233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:59637)
0929 18:44:52.313703 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:52.313814 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":121,"threads_started":1}
W20260929 18:44:52.315447 23103 heartbeater.cc:644] Failed to heartbeat to 127.7.157.126:32875 (0 consecutive failures): Network error: Failed to ping master at 127.7.157.126:32875: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
I20260929 18:44:52.316119 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:52.315813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:38387)
0929 18:44:52.315922 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:52.316059 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 18:44:52.316728 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:52.316531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:53065)
0929 18:44:52.316632 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:52.316679 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 18:44:52.316936 23103 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:44:52.317147 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 22974
I20260929 18:44:53.024677 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:53.024230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:42193)
0929 18:44:53.024497 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:53.024562 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
I20260929 18:44:53.122484 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:53.122042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:42023)
0929 18:44:53.122313 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:53.122380 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260929 18:44:53.317884 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:53.317479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:60547)
0929 18:44:53.317723 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:53.317783 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260929 18:44:53.445643 23110 raft_consensus.cc:482] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:44:53.445827 23110 raft_consensus.cc:504] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:53.446889 23110 leader_election.cc:288] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791), e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663)
Sep 29 18:44:53 dist-test-slave-smpn krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707491, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.66@KRBTEST.COM
Sep 29 18:44:53 dist-test-slave-smpn krb5kdc[21530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707491, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.67@KRBTEST.COM
I20260929 18:44:53.463191 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:53.447301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:36573 (local address 127.7.157.66:39663)
0929 18:44:53.447635 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:53.447658 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:53.447697 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:44:53.448549 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:53.448558 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:53.449012 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:53.449347 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:53.450979 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:53.452557 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:53.454516 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:53.455281 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:53.455447 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:53.458634 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:53.458640 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:53.458654 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:53.458684 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:53.460483 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:53.461181 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:53.461187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:53.461193 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:53.461308 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:53.461659 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:53.461662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:53.461664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:53.462056 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:53.462166 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:53.462714 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 18:44:53.462982 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260929 18:44:53.463193 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:53.447294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:42675 (local address 127.7.157.67:33791)
0929 18:44:53.447708 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 18:44:53.447734 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 18:44:53.447792 (+ 58us) server_negotiation.cc:407] Connection header received
0929 18:44:53.448685 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:44:53.448691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:44:53.449012 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:44:53.449347 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:44:53.450972 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:53.452551 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:53.454516 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:44:53.455281 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:44:53.455447 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:53.458832 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:44:53.458837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:44:53.458850 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:44:53.458879 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:44:53.460616 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:53.461175 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:53.461181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:53.461185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:53.461300 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:44:53.461640 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:44:53.461643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:44:53.461645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:44:53.462063 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:44:53.462172 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:44:53.462711 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 18:44:53.462970 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":111,"threads_started":1}
I20260929 18:44:53.463182 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:53.447111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:33791 (local address 127.7.157.65:42675)
0929 18:44:53.447701 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:53.447715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:53.447852 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:53.449704 (+ 1852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:53.449707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:53.450010 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:53.450787 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:53.450808 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:53.452732 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:53.452735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:53.454403 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:53.454414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:53.455432 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:53.455437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:53.455754 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:53.456452 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:53.456480 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:53.458649 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:53.460713 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:53.460716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:53.460718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:53.461063 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:53.461401 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:53.461403 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:53.461404 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:53.461540 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:53.462179 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:53.462186 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:53.462413 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 18:44:53.462576 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 18:44:53.462951 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":588,"spinlock_wait_cycles":28800,"thread_start_us":83,"threads_started":1}
I20260929 18:44:53.463194 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:53.447109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:39663 (local address 127.7.157.65:36573)
0929 18:44:53.447234 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:53.447248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:44:53.447340 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:44:53.449447 (+ 2107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:44:53.449453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:44:53.450004 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:44:53.450787 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:53.450808 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:53.452734 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:53.452737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:53.454403 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:44:53.454414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:44:53.455432 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:44:53.455437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:44:53.455724 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:44:53.456452 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:44:53.456480 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:44:53.458383 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:44:53.460634 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:53.460641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:53.460655 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:53.461072 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:53.461410 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:44:53.461413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:44:53.461416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:44:53.461566 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:44:53.462176 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:44:53.462183 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:44:53.462413 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 18:44:53.462580 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 18:44:53.463026 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":577,"spinlock_wait_cycles":26112}
W20260929 18:44:53.465828 22723 leader_election.cc:341] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663): Not found: Tablet not found: 704c83a269674ec4a7986768eea013fa
W20260929 18:44:53.466044 22725 leader_election.cc:341] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791): Not found: Tablet not found: 704c83a269674ec4a7986768eea013fa
I20260929 18:44:53.466112 22725 leader_election.cc:302] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7a318f8ef3449feac4b8e87cf155dce; no voters: 7475bf8492804c1593b1bda97b68e5be, e82a933a7ee14b64923c412b6372d817
I20260929 18:44:53.466332 23110 raft_consensus.cc:2726] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:44:54.025806 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:54.025418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:54197)
0929 18:44:54.025655 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:54.025709 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260929 18:44:54.123605 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:54.123200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:55869)
0929 18:44:54.123452 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:54.123509 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":65,"threads_started":1}
I20260929 18:44:54.319152 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:54.318655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:49145)
0929 18:44:54.318956 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:54.319018 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":80,"threads_started":1}
I20260929 18:44:55.027115 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:55.026693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:59897)
0929 18:44:55.026929 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:55.026984 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260929 18:44:55.092425 23118 raft_consensus.cc:482] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:44:55.092557 23118 raft_consensus.cc:504] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:55.092940 23118 leader_election.cc:288] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791), e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663)
W20260929 18:44:55.093736 22723 leader_election.cc:341] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663): Not found: Tablet not found: 704c83a269674ec4a7986768eea013fa
W20260929 18:44:55.093839 22725 leader_election.cc:341] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791): Not found: Tablet not found: 704c83a269674ec4a7986768eea013fa
I20260929 18:44:55.093940 22725 leader_election.cc:302] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7a318f8ef3449feac4b8e87cf155dce; no voters: 7475bf8492804c1593b1bda97b68e5be, e82a933a7ee14b64923c412b6372d817
I20260929 18:44:55.094084 23118 raft_consensus.cc:2726] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:44:55.124747 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:55.124323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:39359)
0929 18:44:55.124584 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:55.124636 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":70,"threads_started":1}
I20260929 18:44:55.320418 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:55.319963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:39363)
0929 18:44:55.320235 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:55.320302 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
I20260929 18:44:56.028357 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:56.027890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:45585)
0929 18:44:56.028188 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:56.028256 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":101,"threads_started":1}
I20260929 18:44:56.125968 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:56.125533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:37075)
0929 18:44:56.125797 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:56.125869 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
I20260929 18:44:56.321686 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:56.321245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:33391)
0929 18:44:56.321530 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:56.321592 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260929 18:44:57.029706 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:57.029219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:51717)
0929 18:44:57.029544 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:57.029605 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":87,"threads_started":1}
I20260929 18:44:57.127210 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:57.126744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:49481)
0929 18:44:57.127013 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:57.127101 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":74,"threads_started":1}
I20260929 18:44:57.322914 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:57.322499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:44871)
0929 18:44:57.322750 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:57.322809 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
I20260929 18:44:58.030973 23127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:58.030510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:55237)
0929 18:44:58.030774 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:58.030838 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260929 18:44:58.128410 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:58.127947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:60249)
0929 18:44:58.128247 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:58.128314 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":71,"threads_started":1}
I20260929 18:44:58.296386 23129 raft_consensus.cc:482] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:44:58.296532 23129 raft_consensus.cc:504] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:44:58.297020 23129 leader_election.cc:288] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791), e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663)
W20260929 18:44:58.297832 22723 leader_election.cc:341] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663): Not found: Tablet not found: 704c83a269674ec4a7986768eea013fa
W20260929 18:44:58.297956 22725 leader_election.cc:341] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791): Not found: Tablet not found: 704c83a269674ec4a7986768eea013fa
I20260929 18:44:58.298053 22725 leader_election.cc:302] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7a318f8ef3449feac4b8e87cf155dce; no voters: 7475bf8492804c1593b1bda97b68e5be, e82a933a7ee14b64923c412b6372d817
I20260929 18:44:58.298251 23129 raft_consensus.cc:2726] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:44:58.324157 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:58.323690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:40637)
0929 18:44:58.323986 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:58.324056 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":79,"threads_started":1}
I20260929 18:44:59.032155 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:59.031697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:60437)
0929 18:44:59.031977 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:59.032036 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":93,"threads_started":1}
I20260929 18:44:59.129648 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:59.129167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:41539)
0929 18:44:59.129470 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:59.129532 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":73,"threads_started":1}
I20260929 18:44:59.325397 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:44:59.324878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:53095)
0929 18:44:59.325237 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 18:44:59.325302 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":86,"threads_started":1}
I20260929 18:45:00.033396 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:00.032991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:50395)
0929 18:45:00.033242 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:00.033299 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
I20260929 18:45:00.130928 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:00.130442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:60569)
0929 18:45:00.130765 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:00.130829 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
I20260929 18:45:00.326467 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:00.326068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:37529)
0929 18:45:00.326312 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:00.326373 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
I20260929 18:45:01.034451 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:01.034084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:37443)
0929 18:45:01.034304 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:01.034355 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20260929 18:45:01.115447 23138 raft_consensus.cc:482] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:45:01.115600 23138 raft_consensus.cc:504] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:45:01.116070 23138 leader_election.cc:288] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791), e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663)
W20260929 18:45:01.116855 22723 leader_election.cc:341] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663): Not found: Tablet not found: 704c83a269674ec4a7986768eea013fa
W20260929 18:45:01.116971 22725 leader_election.cc:341] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791): Not found: Tablet not found: 704c83a269674ec4a7986768eea013fa
I20260929 18:45:01.117063 22725 leader_election.cc:302] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7a318f8ef3449feac4b8e87cf155dce; no voters: 7475bf8492804c1593b1bda97b68e5be, e82a933a7ee14b64923c412b6372d817
I20260929 18:45:01.117216 23138 raft_consensus.cc:2726] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:45:01.132107 23139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:01.131680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:42393)
0929 18:45:01.131932 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:01.131991 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260929 18:45:01.327734 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:01.327335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:40413)
0929 18:45:01.327583 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:01.327641 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260929 18:45:02.035769 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:02.035322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:56139)
0929 18:45:02.035582 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:02.035645 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":73,"threads_started":1}
I20260929 18:45:02.133419 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:02.132960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:44601)
0929 18:45:02.133232 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:02.133299 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20260929 18:45:02.328953 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:02.328504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:55515)
0929 18:45:02.328793 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:02.328854 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":98,"threads_started":1}
I20260929 18:45:03.036975 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:03.036524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:40301)
0929 18:45:03.036810 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:03.036876 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":83,"threads_started":1}
I20260929 18:45:03.134562 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:03.134150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:33489)
0929 18:45:03.134407 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:03.134463 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":71,"threads_started":1}
I20260929 18:45:03.330102 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:03.329701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:53795)
0929 18:45:03.329954 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:03.330006 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":89,"threads_started":1}
I20260929 18:45:04.038138 23147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:04.037765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:42215)
0929 18:45:04.037989 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:04.038041 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260929 18:45:04.135661 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:04.135267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:54785)
0929 18:45:04.135512 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:04.135564 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
I20260929 18:45:04.331255 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:04.330815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:37691)
0929 18:45:04.331099 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:04.331158 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":94,"threads_started":1}
I20260929 18:45:05.039283 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:05.038856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:53853)
0929 18:45:05.039124 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:05.039184 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
I20260929 18:45:05.136742 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:05.136345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:45243)
0929 18:45:05.136592 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:05.136648 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
I20260929 18:45:05.332369 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:05.331969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:59633)
0929 18:45:05.332207 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:05.332264 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260929 18:45:06.040359 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:06.039971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:55709)
0929 18:45:06.040191 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:06.040245 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
I20260929 18:45:06.137970 23154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:06.137515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:42225)
0929 18:45:06.137797 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:06.137860 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":100,"threads_started":1}
I20260929 18:45:06.333731 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:06.333212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.67:51041)
0929 18:45:06.333565 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:06.333630 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":102,"threads_started":1}
I20260929 18:45:06.453084 23156 raft_consensus.cc:482] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:45:06.453217 23156 raft_consensus.cc:504] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7475bf8492804c1593b1bda97b68e5be" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 33791 } } peers { permanent_uuid: "e82a933a7ee14b64923c412b6372d817" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 39663 } } peers { permanent_uuid: "c7a318f8ef3449feac4b8e87cf155dce" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41593 } }
I20260929 18:45:06.453649 23156 leader_election.cc:288] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791), e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663)
W20260929 18:45:06.454355 22723 leader_election.cc:341] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e82a933a7ee14b64923c412b6372d817 (127.7.157.66:39663): Not found: Tablet not found: 704c83a269674ec4a7986768eea013fa
W20260929 18:45:06.454464 22725 leader_election.cc:341] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7475bf8492804c1593b1bda97b68e5be (127.7.157.67:33791): Not found: Tablet not found: 704c83a269674ec4a7986768eea013fa
I20260929 18:45:06.454551 22725 leader_election.cc:302] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7a318f8ef3449feac4b8e87cf155dce; no voters: 7475bf8492804c1593b1bda97b68e5be, e82a933a7ee14b64923c412b6372d817
I20260929 18:45:06.454684 23156 raft_consensus.cc:2726] T 704c83a269674ec4a7986768eea013fa P c7a318f8ef3449feac4b8e87cf155dce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:45:07.041693 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:07.041252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.65:46239)
0929 18:45:07.041529 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:07.041588 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
I20260929 18:45:07.139292 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:07.138835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:32875 (local address 127.7.157.66:45339)
0929 18:45:07.139147 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:07.139196 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
I20260929 18:45:07.323859 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 22673
I20260929 18:45:07.328941 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 22840
I20260929 18:45:07.333691 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 22974
2026-09-29T18:45:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18783 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:45:07 dist-test-slave-smpn krb5kdc[23161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:45:07 dist-test-slave-smpn krb5kdc[23161](info): set up 2 sockets
Sep 29 18:45:07 dist-test-slave-smpn krb5kdc[23161](info): commencing operation
krb5kdc: starting...
W20260929 18:45:08.291338 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:45:08 dist-test-slave-smpn krb5kdc[23161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:45:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:45:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:45:08.409991 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:41607
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:41607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:45:08.517556 23174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:45:08.517853 23174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:45:08.517910 23174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:45:08.521944 23174 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:45:08.522017 23174 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:45:08.522037 23174 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:45:08.522054 23174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:45:08.522078 23174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:45:08.522101 23174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:45:08.522140 23174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:45:08.522171 23174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:45:08.527678 23174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:41607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:41607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:45:08.528797 23174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:45:08.529834 23174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:45:08.538254 23174 server_base.cc:1034] running on GCE node
W20260929 18:45:08.538380 23180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:08.538508 23182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:08.538336 23179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:08.539333 23174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:45:08.540478 23174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:45:08.541673 23174 hybrid_clock.cc:648] HybridClock initialized: now 1790707508541651 us; error 32 us; skew 500 ppm
Sep 29 18:45:08 dist-test-slave-smpn krb5kdc[23161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707508, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:45:08.544783 23174 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:45:08.545885 23174 webserver.cc:460] Webserver started at http://127.7.157.126:42341/ using document root <none> and password file <none>
I20260929 18:45:08.546597 23174 fs_manager.cc:359] Metadata directory not provided
I20260929 18:45:08.546684 23174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:45:08.546948 23174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:45:08.549177 23174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "999b21f5f102427a9127b14ed9d5bd4f"
format_stamp: "Formatted at 2026-09-29 18:45:08 on dist-test-slave-smpn"
server_key: "519648802149821875a642318975c2d8"
server_key_iv: "06320bd8a11792e1cfd67f72014d1f70"
server_key_version: "encryptionkey@0"
I20260929 18:45:08.549747 23174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "999b21f5f102427a9127b14ed9d5bd4f"
format_stamp: "Formatted at 2026-09-29 18:45:08 on dist-test-slave-smpn"
server_key: "519648802149821875a642318975c2d8"
server_key_iv: "06320bd8a11792e1cfd67f72014d1f70"
server_key_version: "encryptionkey@0"
I20260929 18:45:08.553551 23174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:45:08.556099 23189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:08.557426 23174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:45:08.557539 23174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "999b21f5f102427a9127b14ed9d5bd4f"
format_stamp: "Formatted at 2026-09-29 18:45:08 on dist-test-slave-smpn"
server_key: "519648802149821875a642318975c2d8"
server_key_iv: "06320bd8a11792e1cfd67f72014d1f70"
server_key_version: "encryptionkey@0"
I20260929 18:45:08.557649 23174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:45:08.567345 23174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:45:08.570289 23174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:45:08.570469 23174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:45:08.577045 23174 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:41607
I20260929 18:45:08.577282 23241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:41607 every 8 connection(s)
I20260929 18:45:08.577989 23174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:45:08.580747 23242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:45:08.585515 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 23174
I20260929 18:45:08.585625 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:45:08.585867 7797 external_mini_cluster.cc:1428] Setting key 7bbc62aa0b63a8325f8c681ba35fe8f2
I20260929 18:45:08.586282 23242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Bootstrap starting.
I20260929 18:45:08.588860 23242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Neither blocks nor log segments found. Creating new log.
I20260929 18:45:08.589733 23242 log.cc:824] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:45:08.591544 23242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: No bootstrap required, opened a new log
Sep 29 18:45:08 dist-test-slave-smpn krb5kdc[23161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:45:08.595237 23242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } }
I20260929 18:45:08.595491 23242 raft_consensus.cc:374] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:45:08.595549 23242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999b21f5f102427a9127b14ed9d5bd4f, State: Initialized, Role: FOLLOWER
I20260929 18:45:08.596031 23242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } }
I20260929 18:45:08.596215 23242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:45:08.596282 23242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:45:08.596378 23242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:45:08.597374 23242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } }
I20260929 18:45:08.597710 23242 leader_election.cc:302] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 999b21f5f102427a9127b14ed9d5bd4f; no voters:
I20260929 18:45:08.598033 23242 leader_election.cc:288] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:45:08.598114 23247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:45:08.598325 23247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 1 LEADER]: Becoming Leader. State: Replica: 999b21f5f102427a9127b14ed9d5bd4f, State: Running, Role: LEADER
I20260929 18:45:08.598636 23247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } }
I20260929 18:45:08.599346 23242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:45:08.599916 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 999b21f5f102427a9127b14ed9d5bd4f. Latest consensus state: current_term: 1 leader_uuid: "999b21f5f102427a9127b14ed9d5bd4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } } }
I20260929 18:45:08.600024 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [sys.catalog]: This master's current role is: LEADER
I20260929 18:45:08.600207 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:08.587276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.7.157.126:41607)
0929 18:45:08.587690 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:08.587708 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:08.587751 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:45:08.588534 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:08.588558 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:08.588989 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:08.589352 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:45:08.589958 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:08.590971 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:08.591862 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:08.592320 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:08.592477 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:08.595004 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:45:08.595010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:45:08.595025 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:45:08.595055 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:45:08.596973 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:08.597360 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:08.597366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:08.597371 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:08.597507 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:08.597740 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:08.597743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:08.597744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:08.598264 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:45:08.598451 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:08.598885 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 18:45:08.599205 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":116,"threads_started":1}
I20260929 18:45:08.600903 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "999b21f5f102427a9127b14ed9d5bd4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } } }
I20260929 18:45:08.601632 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [sys.catalog]: This master's current role is: LEADER
W20260929 18:45:08.604436 23262 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:45:08.604516 23262 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:45:08.604588 23254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:45:08.605432 23254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:45:08.610883 23254 catalog_manager.cc:1347] Generated new cluster ID: b70889f9c38a414b97993279027d9766
I20260929 18:45:08.610947 23254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:45:08.637594 23254 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:45:08.638594 23254 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:45:08.643725 23254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Generated new TSK 0
I20260929 18:45:08.644443 23254 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:45:08.676386 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:41607
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:45:08.794219 23268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:45:08.794497 23268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:45:08.794538 23268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:45:08.798591 23268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:45:08.798663 23268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:45:08.798722 23268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:45:08.804142 23268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:41607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:45:08.805119 23268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:45:08.806063 23268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:45:08.815055 23274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:08.815058 23273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:08.815259 23268 server_base.cc:1034] running on GCE node
W20260929 18:45:08.815184 23276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:08.815727 23268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:45:08.816385 23268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:45:08.817533 23268 hybrid_clock.cc:648] HybridClock initialized: now 1790707508817511 us; error 39 us; skew 500 ppm
Sep 29 18:45:08 dist-test-slave-smpn krb5kdc[23161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707508, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:45:08.820128 23268 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:45:08.820933 23268 webserver.cc:460] Webserver started at http://127.7.157.65:34707/ using document root <none> and password file <none>
I20260929 18:45:08.821467 23268 fs_manager.cc:359] Metadata directory not provided
I20260929 18:45:08.821527 23268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:45:08.821713 23268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:45:08.823734 23268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "74a874820d3f425a81722abd8c246f2e"
format_stamp: "Formatted at 2026-09-29 18:45:08 on dist-test-slave-smpn"
server_key: "4037efa49659c2a62d778cc97877858c"
server_key_iv: "2e264b41d7c8fbd031bfe2c679542c1a"
server_key_version: "encryptionkey@0"
I20260929 18:45:08.824256 23268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "74a874820d3f425a81722abd8c246f2e"
format_stamp: "Formatted at 2026-09-29 18:45:08 on dist-test-slave-smpn"
server_key: "4037efa49659c2a62d778cc97877858c"
server_key_iv: "2e264b41d7c8fbd031bfe2c679542c1a"
server_key_version: "encryptionkey@0"
I20260929 18:45:08.827790 23268 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:45:08.830049 23283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:08.831251 23268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 18:45:08.831357 23268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "74a874820d3f425a81722abd8c246f2e"
format_stamp: "Formatted at 2026-09-29 18:45:08 on dist-test-slave-smpn"
server_key: "4037efa49659c2a62d778cc97877858c"
server_key_iv: "2e264b41d7c8fbd031bfe2c679542c1a"
server_key_version: "encryptionkey@0"
I20260929 18:45:08.831449 23268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:45:08.856907 23268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:45:08.859766 23268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:45:08.859949 23268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:45:08.860534 23268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:45:08.861443 23268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:45:08.861491 23268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:08.861544 23268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:45:08.861573 23268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:08.870185 23268 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:40143
I20260929 18:45:08.870214 23396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:40143 every 8 connection(s)
I20260929 18:45:08.871055 23268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:45:08.873126 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 23268
I20260929 18:45:08.873234 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:45:08.873493 7797 external_mini_cluster.cc:1428] Setting key 6a1dc58ebc73e88c075da6e3525dafa6
Sep 29 18:45:08 dist-test-slave-smpn krb5kdc[23161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707508, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
I20260929 18:45:08.884574 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:08.872822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:32895 (local address 127.7.157.126:41607)
0929 18:45:08.872960 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:08.872965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:08.873815 (+ 850us) server_negotiation.cc:407] Connection header received
0929 18:45:08.875227 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:08.875230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:08.875285 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:08.875363 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:45:08.876664 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:08.877196 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:08.878345 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:08.878575 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:08.878661 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:08.881204 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:45:08.881207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:45:08.881210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:45:08.881234 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:45:08.882592 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:08.883102 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:08.883105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:08.883107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:08.883186 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:08.883509 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:08.883512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:08.883514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:08.883702 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:45:08.883795 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:08.884273 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 18:45:08.884409 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:45:08.885378 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:08.873201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:32895)
0929 18:45:08.873663 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:08.873715 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:08.874971 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:08.875586 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:08.875593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:08.876017 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:45:08.876520 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:08.876534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:08.877316 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:08.877319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:08.878227 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:08.878236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:08.878711 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:08.878715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:08.878883 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:08.879483 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:45:08.879501 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:45:08.881037 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:45:08.882714 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:08.882720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:08.882736 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:08.882987 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:08.883270 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:08.883273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:08.883275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:08.883422 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:08.883799 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:45:08.883806 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:45:08.884046 (+ 240us) client_negotiation.cc:769] Sending connection context
0929 18:45:08.884215 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 18:45:08.884573 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
I20260929 18:45:08.886756 23397 heartbeater.cc:347] Connected to a master server at 127.7.157.126:41607
I20260929 18:45:08.886991 23397 heartbeater.cc:464] Registering TS with master...
I20260929 18:45:08.887480 23397 heartbeater.cc:511] Master 127.7.157.126:41607 requested a full tablet report, sending...
I20260929 18:45:08.889014 23203 ts_manager.cc:194] Registered new tserver with Master: 74a874820d3f425a81722abd8c246f2e (127.7.157.65:40143)
I20260929 18:45:08.890580 23203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:32895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:45:08.901635 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:41607
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:45:09.008602 23402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:45:09.008868 23402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:45:09.008909 23402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:45:09.012977 23402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:45:09.013039 23402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:45:09.013103 23402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:45:09.018338 23402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:41607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:45:09.019438 23402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:45:09.020418 23402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:45:09.029631 23407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:09.029673 23410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:09.029688 23402 server_base.cc:1034] running on GCE node
W20260929 18:45:09.029640 23408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:09.030265 23402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:45:09.030898 23402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:45:09.032047 23402 hybrid_clock.cc:648] HybridClock initialized: now 1790707509032037 us; error 60 us; skew 500 ppm
Sep 29 18:45:09 dist-test-slave-smpn krb5kdc[23161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707509, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:45:09.034617 23402 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:45:09.035531 23402 webserver.cc:460] Webserver started at http://127.7.157.66:39375/ using document root <none> and password file <none>
I20260929 18:45:09.036053 23402 fs_manager.cc:359] Metadata directory not provided
I20260929 18:45:09.036116 23402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:45:09.036311 23402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:45:09.038323 23402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "446764e8d4d44e90b7744b486190fec0"
format_stamp: "Formatted at 2026-09-29 18:45:09 on dist-test-slave-smpn"
server_key: "4de2ee2c0e0c294b4c2f2c34bb213439"
server_key_iv: "5be3bd01891026995776cf0bb44bef22"
server_key_version: "encryptionkey@0"
I20260929 18:45:09.038848 23402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "446764e8d4d44e90b7744b486190fec0"
format_stamp: "Formatted at 2026-09-29 18:45:09 on dist-test-slave-smpn"
server_key: "4de2ee2c0e0c294b4c2f2c34bb213439"
server_key_iv: "5be3bd01891026995776cf0bb44bef22"
server_key_version: "encryptionkey@0"
I20260929 18:45:09.042481 23402 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:45:09.044839 23417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:09.046008 23402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 18:45:09.046108 23402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "446764e8d4d44e90b7744b486190fec0"
format_stamp: "Formatted at 2026-09-29 18:45:09 on dist-test-slave-smpn"
server_key: "4de2ee2c0e0c294b4c2f2c34bb213439"
server_key_iv: "5be3bd01891026995776cf0bb44bef22"
server_key_version: "encryptionkey@0"
I20260929 18:45:09.046205 23402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:45:09.068188 23402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:45:09.071060 23402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:45:09.071267 23402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:45:09.071858 23402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:45:09.072693 23402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:45:09.072741 23402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:09.072793 23402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:45:09.072820 23402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:09.081055 23402 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:46149
I20260929 18:45:09.081106 23530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:46149 every 8 connection(s)
I20260929 18:45:09.081943 23402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:45:09.087356 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 23402
I20260929 18:45:09.087457 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:45:09.087705 7797 external_mini_cluster.cc:1428] Setting key 67c8c406242603616605061e910b1e13
Sep 29 18:45:09 dist-test-slave-smpn krb5kdc[23161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707509, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:45:09.095722 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.083661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:36527 (local address 127.7.157.126:41607)
0929 18:45:09.083801 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:09.083805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:09.084646 (+ 841us) server_negotiation.cc:407] Connection header received
0929 18:45:09.085901 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:09.085904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:09.085965 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:09.086036 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:45:09.087275 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.087879 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.089162 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.089386 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.089474 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:09.092046 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:45:09.092050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:45:09.092052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:45:09.092077 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:45:09.093500 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:09.094118 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:09.094120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:09.094121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:09.094197 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:09.094543 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:09.094545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:09.094546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:09.094724 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:45:09.094795 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:09.095464 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 18:45:09.095570 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
I20260929 18:45:09.096923 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.084081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:36527)
0929 18:45:09.084503 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:09.084544 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:09.085683 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:09.086231 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:09.086239 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:09.086614 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:45:09.087132 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:09.087149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.087998 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.088002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:09.089044 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:09.089056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.089524 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.089527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:09.089698 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:09.090269 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:45:09.090300 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:45:09.091885 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:45:09.093619 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:09.093625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:09.093643 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:09.094024 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:09.094276 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:09.094279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:09.094281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:09.094463 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:09.094822 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:45:09.094830 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:45:09.095202 (+ 372us) client_negotiation.cc:769] Sending connection context
0929 18:45:09.095430 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 18:45:09.095804 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
I20260929 18:45:09.098104 23531 heartbeater.cc:347] Connected to a master server at 127.7.157.126:41607
I20260929 18:45:09.098323 23531 heartbeater.cc:464] Registering TS with master...
I20260929 18:45:09.098767 23531 heartbeater.cc:511] Master 127.7.157.126:41607 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
I20260929 18:45:09.099885 23203 ts_manager.cc:194] Registered new tserver with Master: 446764e8d4d44e90b7744b486190fec0 (127.7.157.66:46149)
I20260929 18:45:09.100529 23203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:36527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:45:09.114542 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:41607
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:45:09.223990 23536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:45:09.224267 23536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:45:09.224311 23536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:45:09.228364 23536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:45:09.228430 23536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:45:09.228504 23536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:45:09.233892 23536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:41607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:45:09.235050 23536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:45:09.236184 23536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:45:09.245774 23542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:09.245795 23544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:09.245802 23541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:09.246052 23536 server_base.cc:1034] running on GCE node
I20260929 18:45:09.246491 23536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:45:09.247208 23536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:45:09.248368 23536 hybrid_clock.cc:648] HybridClock initialized: now 1790707509248338 us; error 45 us; skew 500 ppm
Sep 29 18:45:09 dist-test-slave-smpn krb5kdc[23161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707509, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:45:09.251168 23536 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:45:09.252048 23536 webserver.cc:460] Webserver started at http://127.7.157.67:45739/ using document root <none> and password file <none>
I20260929 18:45:09.252593 23536 fs_manager.cc:359] Metadata directory not provided
I20260929 18:45:09.252660 23536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:45:09.252872 23536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:45:09.254961 23536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "1567cd8458dc4a1db7f421c9f11417d7"
format_stamp: "Formatted at 2026-09-29 18:45:09 on dist-test-slave-smpn"
server_key: "85334c29ea1a6e4b2f7936017497446d"
server_key_iv: "6a38a04ce638ac398f4a6f5c40d920e7"
server_key_version: "encryptionkey@0"
I20260929 18:45:09.255550 23536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "1567cd8458dc4a1db7f421c9f11417d7"
format_stamp: "Formatted at 2026-09-29 18:45:09 on dist-test-slave-smpn"
server_key: "85334c29ea1a6e4b2f7936017497446d"
server_key_iv: "6a38a04ce638ac398f4a6f5c40d920e7"
server_key_version: "encryptionkey@0"
I20260929 18:45:09.259336 23536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:45:09.261957 23551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:09.263305 23536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:45:09.263427 23536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "1567cd8458dc4a1db7f421c9f11417d7"
format_stamp: "Formatted at 2026-09-29 18:45:09 on dist-test-slave-smpn"
server_key: "85334c29ea1a6e4b2f7936017497446d"
server_key_iv: "6a38a04ce638ac398f4a6f5c40d920e7"
server_key_version: "encryptionkey@0"
I20260929 18:45:09.263532 23536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:45:09.279737 23536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:45:09.282974 23536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:45:09.283203 23536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:45:09.283820 23536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:45:09.284662 23536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:45:09.284709 23536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:09.284762 23536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:45:09.284797 23536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:09.293156 23536 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:35489
I20260929 18:45:09.293179 23664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:35489 every 8 connection(s)
I20260929 18:45:09.294075 23536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:45:09.300581 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 23536
I20260929 18:45:09.300702 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:45:09.301019 7797 external_mini_cluster.cc:1428] Setting key af196603c030446105531c2b5ebd6e47
Sep 29 18:45:09 dist-test-slave-smpn krb5kdc[23161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707509, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:45:09.307911 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.295990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:54091 (local address 127.7.157.126:41607)
0929 18:45:09.296181 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:09.296185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:09.297013 (+ 828us) server_negotiation.cc:407] Connection header received
0929 18:45:09.298232 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:09.298235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:09.298286 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:09.298348 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:45:09.299636 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.300233 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.301707 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.301983 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.302082 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:09.304566 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:45:09.304569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:45:09.304571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:45:09.304597 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:45:09.305877 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:09.306353 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:09.306355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:09.306357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:09.306434 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:09.306831 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:09.306832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:09.306834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:09.307009 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:45:09.307107 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:09.307677 (+ 570us) server_negotiation.cc:299] Negotiation successful
0929 18:45:09.307790 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 18:45:09.308802 23667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.296364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:54091)
0929 18:45:09.296859 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:09.296903 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:09.298026 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:09.298512 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:09.298519 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:09.298918 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:45:09.299506 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:09.299521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.300353 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.300356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:09.301589 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:09.301600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.302127 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.302131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:09.302330 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:09.302898 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:45:09.302919 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:45:09.304405 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:45:09.305991 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:09.305996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:09.306010 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:09.306274 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:09.306538 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:09.306541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:09.306543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:09.306743 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:09.307140 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:45:09.307148 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:45:09.307429 (+ 281us) client_negotiation.cc:769] Sending connection context
0929 18:45:09.307652 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 18:45:09.307936 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":100,"threads_started":1}
I20260929 18:45:09.309945 23665 heartbeater.cc:347] Connected to a master server at 127.7.157.126:41607
I20260929 18:45:09.310194 23665 heartbeater.cc:464] Registering TS with master...
I20260929 18:45:09.310708 23665 heartbeater.cc:511] Master 127.7.157.126:41607 requested a full tablet report, sending...
I20260929 18:45:09.311784 23203 ts_manager.cc:194] Registered new tserver with Master: 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489)
I20260929 18:45:09.312366 23203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:54091
I20260929 18:45:09.314814 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:45:09.323683 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.316667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.7.157.126:41607)
0929 18:45:09.316777 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:09.316790 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:09.316950 (+ 160us) server_negotiation.cc:407] Connection header received
0929 18:45:09.317076 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:09.317079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:09.317124 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:09.317181 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:45:09.317846 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.318392 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.319255 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.319473 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.319557 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:09.320853 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:45:09.320857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:45:09.320859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:45:09.320884 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:45:09.322069 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:09.322472 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:09.322475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:09.322477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:09.322555 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:09.322804 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:09.322807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:09.322809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:09.323038 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:45:09.323190 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:09.323409 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 18:45:09.323524 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:45:09.326413 23203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:45:09.329691 23203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:45:09.343681 23677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.337995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60750 (local address 127.7.157.65:40143)
0929 18:45:09.338900 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:09.338915 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:09.338937 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:45:09.339015 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:09.339020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:09.339262 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:09.339454 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:45:09.339728 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.340620 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.342186 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.342983 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.343122 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:09.343280 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:09.343430 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 18:45:09.343530 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":96,"threads_started":1}
I20260929 18:45:09.343669 23676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.337914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:40143 (local address 127.0.0.1:60750)
0929 18:45:09.338287 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:09.338341 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:09.338475 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:09.339452 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:09.339458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:09.339482 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:45:09.339619 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:09.339626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.340739 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.340743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:09.342054 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:09.342062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.343126 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.343130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:09.343233 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:09.343269 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 18:45:09.343387 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 18:45:09.343512 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":20,"thread_start_us":98,"threads_started":1}
I20260929 18:45:09.343884 23678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.338224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:35489 (local address 127.0.0.1:36126)
0929 18:45:09.338665 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:09.338679 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:09.338766 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:09.339837 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:09.339840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:09.339856 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:45:09.339943 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:09.339950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.341359 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.341363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:09.342447 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:09.342456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.343486 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.343490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:09.343595 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:09.343617 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:45:09.343676 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 18:45:09.343752 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":26,"thread_start_us":116,"threads_started":1}
I20260929 18:45:09.343995 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.339021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55390 (local address 127.7.157.66:46149)
0929 18:45:09.339466 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:09.339477 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:09.339500 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:45:09.339568 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:09.339573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:09.339768 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:09.339910 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:45:09.340229 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.341341 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.342534 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.343102 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.343198 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:09.343353 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:09.343758 (+ 405us) server_negotiation.cc:299] Negotiation successful
0929 18:45:09.343859 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":128,"threads_started":1}
I20260929 18:45:09.344108 23680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.338651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:46149 (local address 127.0.0.1:55390)
0929 18:45:09.339199 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:09.339211 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:09.339290 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:09.340011 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:09.340014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:09.340031 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:45:09.340106 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:09.340112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.341456 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.341459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:09.342426 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:09.342435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.343411 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.343414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:09.343490 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:09.343542 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 18:45:09.343606 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:45:09.343683 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":25,"thread_start_us":85,"threads_started":1}
I20260929 18:45:09.344866 23679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.338486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.7.157.67:35489)
0929 18:45:09.338963 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:09.338973 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:09.338990 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:45:09.339046 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:09.339050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:09.339696 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:09.339853 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:45:09.340072 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.341215 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.342572 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.343358 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.344441 (+ 1083us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:09.344574 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:09.344622 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 18:45:09.344723 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":93,"threads_started":1}
I20260929 18:45:09.347023 23599 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7640db70534a67bf14fa5081109934 (DEFAULT_TABLE table=test-table [id=238be7f256ef4f219332068f71593198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:45:09.347023 23465 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7640db70534a67bf14fa5081109934 (DEFAULT_TABLE table=test-table [id=238be7f256ef4f219332068f71593198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:45:09.347024 23331 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7640db70534a67bf14fa5081109934 (DEFAULT_TABLE table=test-table [id=238be7f256ef4f219332068f71593198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:45:09.348076 23599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7640db70534a67bf14fa5081109934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:45:09.348090 23331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7640db70534a67bf14fa5081109934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:45:09.348091 23465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7640db70534a67bf14fa5081109934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:45:09.353988 23682 tablet_bootstrap.cc:492] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7: Bootstrap starting.
I20260929 18:45:09.355774 23684 tablet_bootstrap.cc:492] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: Bootstrap starting.
I20260929 18:45:09.356164 23683 tablet_bootstrap.cc:492] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0: Bootstrap starting.
I20260929 18:45:09.356215 23682 tablet_bootstrap.cc:654] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7: Neither blocks nor log segments found. Creating new log.
I20260929 18:45:09.357244 23682 log.cc:824] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:45:09.358352 23683 tablet_bootstrap.cc:654] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0: Neither blocks nor log segments found. Creating new log.
I20260929 18:45:09.358978 23682 tablet_bootstrap.cc:492] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7: No bootstrap required, opened a new log
I20260929 18:45:09.359022 23683 log.cc:824] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:45:09.359202 23682 ts_tablet_manager.cc:1397] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:45:09.360736 23683 tablet_bootstrap.cc:492] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0: No bootstrap required, opened a new log
I20260929 18:45:09.360904 23683 ts_tablet_manager.cc:1397] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:45:09.361485 23684 tablet_bootstrap.cc:654] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: Neither blocks nor log segments found. Creating new log.
I20260929 18:45:09.362738 23684 log.cc:824] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:45:09.363428 23682 raft_consensus.cc:348] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } } peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } }
I20260929 18:45:09.363716 23682 raft_consensus.cc:374] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:45:09.363803 23682 raft_consensus.cc:729] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1567cd8458dc4a1db7f421c9f11417d7, State: Initialized, Role: FOLLOWER
I20260929 18:45:09.364260 23682 consensus_queue.cc:260] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } } peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } }
I20260929 18:45:09.364435 23683 raft_consensus.cc:348] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } } peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } }
I20260929 18:45:09.364751 23683 raft_consensus.cc:374] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:45:09.364842 23683 raft_consensus.cc:729] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446764e8d4d44e90b7744b486190fec0, State: Initialized, Role: FOLLOWER
I20260929 18:45:09.364889 23684 tablet_bootstrap.cc:492] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: No bootstrap required, opened a new log
I20260929 18:45:09.364928 23665 heartbeater.cc:503] Master 127.7.157.126:41607 was elected leader, sending a full tablet report...
I20260929 18:45:09.365120 23684 ts_tablet_manager.cc:1397] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20260929 18:45:09.365252 23683 consensus_queue.cc:260] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } } peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } }
I20260929 18:45:09.365828 23531 heartbeater.cc:503] Master 127.7.157.126:41607 was elected leader, sending a full tablet report...
I20260929 18:45:09.366168 23683 ts_tablet_manager.cc:1428] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:45:09.368819 23684 raft_consensus.cc:348] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } } peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } }
I20260929 18:45:09.369165 23684 raft_consensus.cc:374] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:45:09.369266 23684 raft_consensus.cc:729] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a874820d3f425a81722abd8c246f2e, State: Initialized, Role: FOLLOWER
I20260929 18:45:09.369387 23682 ts_tablet_manager.cc:1428] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260929 18:45:09.369755 23684 consensus_queue.cc:260] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } } peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } }
I20260929 18:45:09.370436 23397 heartbeater.cc:503] Master 127.7.157.126:41607 was elected leader, sending a full tablet report...
I20260929 18:45:09.370774 23684 ts_tablet_manager.cc:1428] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
W20260929 18:45:09.372129 23398 tablet.cc:2369] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:45:09.501777 23689 raft_consensus.cc:482] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:45:09.502022 23689 raft_consensus.cc:504] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } } peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } }
I20260929 18:45:09.503345 23689 leader_election.cc:288] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a874820d3f425a81722abd8c246f2e (127.7.157.65:40143), 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489)
I20260929 18:45:09.507054 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.503553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:35489 (local address 127.7.157.66:50301)
0929 18:45:09.503653 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:09.503666 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:09.503768 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:09.504162 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:09.504165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:09.504188 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:45:09.504325 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:09.504332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.505080 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.505083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:09.505980 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:09.505989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.506627 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.506630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:09.506744 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:09.506779 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 18:45:09.506847 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:45:09.506915 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":71}
I20260929 18:45:09.507054 23677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.503702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:41863 (local address 127.7.157.65:40143)
0929 18:45:09.503834 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:09.503840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:09.504082 (+ 242us) server_negotiation.cc:407] Connection header received
0929 18:45:09.504160 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:09.504163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:09.504213 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:09.504310 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:45:09.504542 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.505016 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.506108 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.506485 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.506585 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:09.506641 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:09.506855 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 18:45:09.506929 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:45:09.507089 23679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.503798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:50301 (local address 127.7.157.67:35489)
0929 18:45:09.503925 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:09.503929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:09.503942 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:45:09.503997 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:09.504000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:09.504046 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:09.504123 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:45:09.504462 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.504974 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.506115 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.506507 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.506615 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:09.506665 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:09.506881 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 18:45:09.506958 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:45:09.507105 23691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.503553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:40143 (local address 127.7.157.66:41863)
0929 18:45:09.503974 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:09.503985 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:09.504071 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:09.504322 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:09.504325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:09.504339 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:45:09.504439 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:09.504445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.505134 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.505136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:09.505977 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:09.505986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.506604 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.506608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:09.506702 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:09.506762 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 18:45:09.506819 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 18:45:09.506894 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":98,"thread_start_us":151,"threads_started":1}
I20260929 18:45:09.507827 23619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7640db70534a67bf14fa5081109934" candidate_uuid: "446764e8d4d44e90b7744b486190fec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1567cd8458dc4a1db7f421c9f11417d7" is_pre_election: true
I20260929 18:45:09.507849 23351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7640db70534a67bf14fa5081109934" candidate_uuid: "446764e8d4d44e90b7744b486190fec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a874820d3f425a81722abd8c246f2e" is_pre_election: true
I20260929 18:45:09.508112 23619 raft_consensus.cc:2454] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446764e8d4d44e90b7744b486190fec0 in term 0.
I20260929 18:45:09.508112 23351 raft_consensus.cc:2454] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446764e8d4d44e90b7744b486190fec0 in term 0.
I20260929 18:45:09.508570 23421 leader_election.cc:302] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 446764e8d4d44e90b7744b486190fec0, 74a874820d3f425a81722abd8c246f2e; no voters:
I20260929 18:45:09.508826 23689 raft_consensus.cc:2781] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:45:09.508899 23689 raft_consensus.cc:482] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:45:09.508927 23689 raft_consensus.cc:3037] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:45:09.509956 23689 raft_consensus.cc:504] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } } peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } }
I20260929 18:45:09.510278 23689 leader_election.cc:288] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [CANDIDATE]: Term 1 election: Requested vote from peers 74a874820d3f425a81722abd8c246f2e (127.7.157.65:40143), 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489)
I20260929 18:45:09.510600 23619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7640db70534a67bf14fa5081109934" candidate_uuid: "446764e8d4d44e90b7744b486190fec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1567cd8458dc4a1db7f421c9f11417d7"
I20260929 18:45:09.510600 23351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7640db70534a67bf14fa5081109934" candidate_uuid: "446764e8d4d44e90b7744b486190fec0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a874820d3f425a81722abd8c246f2e"
I20260929 18:45:09.510725 23619 raft_consensus.cc:3037] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:45:09.510710 23351 raft_consensus.cc:3037] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:45:09.511750 23619 raft_consensus.cc:2454] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446764e8d4d44e90b7744b486190fec0 in term 1.
I20260929 18:45:09.511757 23351 raft_consensus.cc:2454] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446764e8d4d44e90b7744b486190fec0 in term 1.
I20260929 18:45:09.512029 23420 leader_election.cc:302] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1567cd8458dc4a1db7f421c9f11417d7, 446764e8d4d44e90b7744b486190fec0, 74a874820d3f425a81722abd8c246f2e; no voters:
I20260929 18:45:09.512202 23689 raft_consensus.cc:2781] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:45:09.512408 23689 raft_consensus.cc:686] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [term 1 LEADER]: Becoming Leader. State: Replica: 446764e8d4d44e90b7744b486190fec0, State: Running, Role: LEADER
I20260929 18:45:09.512737 23689 consensus_queue.cc:237] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } } peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } }
I20260929 18:45:09.515748 23202 catalog_manager.cc:5495] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 446764e8d4d44e90b7744b486190fec0 (127.7.157.66). New cstate: current_term: 1 leader_uuid: "446764e8d4d44e90b7744b486190fec0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } health_report { overall_health: UNKNOWN } } }
I20260929 18:45:09.534178 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.530751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.7.157.66:46149)
0929 18:45:09.530883 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:09.530887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:09.530901 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:45:09.531039 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:09.531042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:09.531130 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:09.531222 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:45:09.531454 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.531952 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.532785 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:09.533000 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:09.533086 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:09.533373 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:45:09.533783 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:45:09.533891 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:09.533956 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:45:09.534011 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:45:09.539779 23619 raft_consensus.cc:1261] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7 [term 1 FOLLOWER]: Refusing update from remote peer 446764e8d4d44e90b7744b486190fec0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:45:09.539794 23351 raft_consensus.cc:1261] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Refusing update from remote peer 446764e8d4d44e90b7744b486190fec0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:45:09.540443 23689 consensus_queue.cc:1035] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:45:09.540624 23692 consensus_queue.cc:1035] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:45:09.544888 23666 tablet.cc:2369] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:45:09.549647 23697 mvcc.cc:205] Tried to move back new op lower bound from 7334737959068315648 to 7334737958966669312. Current Snapshot: MvccSnapshot[applied={T|T < 7334737959068315648}]
I20260929 18:45:09.549829 23696 mvcc.cc:205] Tried to move back new op lower bound from 7334737959068315648 to 7334737958966669312. Current Snapshot: MvccSnapshot[applied={T|T < 7334737959068315648}]
I20260929 18:45:09.559175 23202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:45:09.559407 23202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:45:09.562062 23202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Sending DeleteTablet for 3 replicas of tablet 3c7640db70534a67bf14fa5081109934
I20260929 18:45:09.562939 23599 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7640db70534a67bf14fa5081109934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:45:09 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:36126
I20260929 18:45:09.563037 23331 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7640db70534a67bf14fa5081109934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:45:09 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:60750
I20260929 18:45:09.563524 23465 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7640db70534a67bf14fa5081109934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:45:09 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:55390
I20260929 18:45:09.563547 23706 tablet_replica.cc:330] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7: stopping tablet replica
I20260929 18:45:09.563920 23706 raft_consensus.cc:2229] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:45:09.563942 23707 tablet_replica.cc:330] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0: stopping tablet replica
I20260929 18:45:09.563994 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 23268
I20260929 18:45:09.564227 23707 raft_consensus.cc:2229] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:45:09.564356 23706 raft_consensus.cc:2258] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:45:09.564600 23707 raft_consensus.cc:2258] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:45:09.566110 23706 ts_tablet_manager.cc:1905] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:45:09.567142 23707 ts_tablet_manager.cc:1905] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:45:09.569259 23706 ts_tablet_manager.cc:1918] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:45:09.569410 23706 log.cc:1196] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/3c7640db70534a67bf14fa5081109934
I20260929 18:45:09.569753 23707 ts_tablet_manager.cc:1918] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:45:09.569800 23706 ts_tablet_manager.cc:1939] T 3c7640db70534a67bf14fa5081109934 P 1567cd8458dc4a1db7f421c9f11417d7: Deleting consensus metadata
I20260929 18:45:09.569895 23707 log.cc:1196] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/3c7640db70534a67bf14fa5081109934
I20260929 18:45:09.570187 23707 ts_tablet_manager.cc:1939] T 3c7640db70534a67bf14fa5081109934 P 446764e8d4d44e90b7744b486190fec0: Deleting consensus metadata
I20260929 18:45:09.571194 23192 catalog_manager.cc:4841] TS 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489): tablet 3c7640db70534a67bf14fa5081109934 (table test-table [id=238be7f256ef4f219332068f71593198]) successfully deleted
I20260929 18:45:09.571473 23193 catalog_manager.cc:4841] TS 446764e8d4d44e90b7744b486190fec0 (127.7.157.66:46149): tablet 3c7640db70534a67bf14fa5081109934 (table test-table [id=238be7f256ef4f219332068f71593198]) successfully deleted
W20260929 18:45:09.572630 23193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:40143 (error 108)
I20260929 18:45:09.572800 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 23402
I20260929 18:45:09.574266 23680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:09.573982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:40143 (local address 127.0.0.1:60766)
0929 18:45:09.574114 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:09.574191 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:40143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 18:45:09.574426 23193 catalog_manager.cc:4584] TS 74a874820d3f425a81722abd8c246f2e (127.7.157.65:40143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c7640db70534a67bf14fa5081109934: Network error: Client connection negotiation failed: client connection to 127.7.157.65:40143: connect: Connection refused (error 111)
I20260929 18:45:09.578658 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 23536
I20260929 18:45:09.583621 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 23174
I20260929 18:45:29.589426 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:41607
--webserver_interface=127.7.157.126
--webserver_port=42341
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:41607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:45:29.697965 23709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:45:29.698231 23709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:45:29.698269 23709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:45:29.702350 23709 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:45:29.702417 23709 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:45:29.702433 23709 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:45:29.702447 23709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:45:29.702464 23709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:45:29.702485 23709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:45:29.702522 23709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:45:29.702549 23709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:45:29.708065 23709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:41607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:41607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=42341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:45:29.709187 23709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:45:29.710235 23709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:45:29.723284 23709 server_base.cc:1034] running on GCE node
W20260929 18:45:29.723220 23716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:29.723227 23718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:29.723222 23715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:29.724037 23709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:45:29.725216 23709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:45:29.726385 23709 hybrid_clock.cc:648] HybridClock initialized: now 1790707529726365 us; error 38 us; skew 500 ppm
Sep 29 18:45:29 dist-test-slave-smpn krb5kdc[23161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707529, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:45:29.734262 23709 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:45:29.735422 23709 webserver.cc:460] Webserver started at http://127.7.157.126:42341/ using document root <none> and password file <none>
I20260929 18:45:29.736065 23709 fs_manager.cc:359] Metadata directory not provided
I20260929 18:45:29.736153 23709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:45:29.740264 23709 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 18:45:29.742467 23725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:29.743937 23709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:45:29.744074 23709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "999b21f5f102427a9127b14ed9d5bd4f"
format_stamp: "Formatted at 2026-09-29 18:45:08 on dist-test-slave-smpn"
server_key: "519648802149821875a642318975c2d8"
server_key_iv: "06320bd8a11792e1cfd67f72014d1f70"
server_key_version: "encryptionkey@0"
I20260929 18:45:29.744720 23709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:45:29.762281 23709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:45:29.765185 23709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:45:29.765399 23709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:45:29.772735 23709 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:41607
I20260929 18:45:29.772987 23777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:41607 every 8 connection(s)
I20260929 18:45:29.774001 23709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:45:29.775686 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 23709
I20260929 18:45:29.776473 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:40143
--local_ip_for_outbound_sockets=127.7.157.65
--tserver_master_addrs=127.7.157.126:41607
--webserver_port=34707
--webserver_interface=127.7.157.65
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:45:29.779225 23778 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:45:29.781478 23778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Bootstrap starting.
I20260929 18:45:29.801811 23778 log.cc:824] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:45:29.810426 23778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:45:29.810798 23778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Bootstrap complete.
I20260929 18:45:29.814634 23778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } }
I20260929 18:45:29.815021 23778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999b21f5f102427a9127b14ed9d5bd4f, State: Initialized, Role: FOLLOWER
I20260929 18:45:29.815578 23778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } }
I20260929 18:45:29.815704 23778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:45:29.815771 23778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:45:29.815864 23778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:45:29.817770 23778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } }
I20260929 18:45:29.818084 23778 leader_election.cc:302] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 999b21f5f102427a9127b14ed9d5bd4f; no voters:
I20260929 18:45:29.818369 23778 leader_election.cc:288] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:45:29.818502 23782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:45:29.818861 23782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 2 LEADER]: Becoming Leader. State: Replica: 999b21f5f102427a9127b14ed9d5bd4f, State: Running, Role: LEADER
I20260929 18:45:29.819257 23782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } }
I20260929 18:45:29.819561 23778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:45:29.820703 23784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 999b21f5f102427a9127b14ed9d5bd4f. Latest consensus state: current_term: 2 leader_uuid: "999b21f5f102427a9127b14ed9d5bd4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } } }
I20260929 18:45:29.820817 23784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [sys.catalog]: This master's current role is: LEADER
I20260929 18:45:29.820760 23783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "999b21f5f102427a9127b14ed9d5bd4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } } }
I20260929 18:45:29.820865 23783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [sys.catalog]: This master's current role is: LEADER
I20260929 18:45:29.824757 23797 catalog_manager.cc:1259] Loaded cluster ID: b70889f9c38a414b97993279027d9766
I20260929 18:45:29.824841 23797 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: loading cluster ID for follower catalog manager: success
I20260929 18:45:29.827478 23797 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: acquiring CA information for follower catalog manager: success
I20260929 18:45:29.829002 23797 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:45:29.829211 23791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:45:29.831777 23791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:45:29.832254 23791 catalog_manager.cc:1259] Loaded cluster ID: b70889f9c38a414b97993279027d9766
I20260929 18:45:29.832312 23791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:45:29.833009 23791 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:45:29.833484 23791 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Loaded TSK: 0
I20260929 18:45:29.847137 23791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Generated new TSK 1
I20260929 18:45:29.847404 23791 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:45:29.895493 23780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:45:29.895753 23780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:45:29.895809 23780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:45:29.899763 23780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:45:29.899818 23780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:45:29.899894 23780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:45:29.905153 23780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:40143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=34707
--tserver_master_addrs=127.7.157.126:41607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:45:29.906143 23780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:45:29.907142 23780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:45:29.917423 23808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:29.917573 23780 server_base.cc:1034] running on GCE node
W20260929 18:45:29.917472 23805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:29.917452 23806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:29.918174 23780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:45:29.918853 23780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:45:29.920020 23780 hybrid_clock.cc:648] HybridClock initialized: now 1790707529920000 us; error 67 us; skew 500 ppm
Sep 29 18:45:29 dist-test-slave-smpn krb5kdc[23161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707529, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:45:29.922991 23780 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:45:29.923882 23780 webserver.cc:460] Webserver started at http://127.7.157.65:34707/ using document root <none> and password file <none>
I20260929 18:45:29.924417 23780 fs_manager.cc:359] Metadata directory not provided
I20260929 18:45:29.924501 23780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:45:29.928392 23780 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:45:29.930687 23815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:29.932047 23780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:45:29.932166 23780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "74a874820d3f425a81722abd8c246f2e"
format_stamp: "Formatted at 2026-09-29 18:45:08 on dist-test-slave-smpn"
server_key: "4037efa49659c2a62d778cc97877858c"
server_key_iv: "2e264b41d7c8fbd031bfe2c679542c1a"
server_key_version: "encryptionkey@0"
I20260929 18:45:29.932729 23780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:45:29.955529 23780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:45:29.958616 23780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:45:29.958804 23780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:45:29.959463 23780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:45:29.960623 23822 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:45:29.964689 23780 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:45:29.964748 23780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260929 18:45:29.964795 23780 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:45:29.966374 23780 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:45:29.966431 23780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 18:45:29.966703 23822 tablet_bootstrap.cc:492] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: Bootstrap starting.
I20260929 18:45:29.976747 23780 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:40143
I20260929 18:45:29.976790 23929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:40143 every 8 connection(s)
I20260929 18:45:29.977998 23780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:45:29.983790 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 23780
I20260929 18:45:29.984287 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:46149
--local_ip_for_outbound_sockets=127.7.157.66
--tserver_master_addrs=127.7.157.126:41607
--webserver_port=39375
--webserver_interface=127.7.157.66
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:45:29.986826 23822 log.cc:824] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:45:29 dist-test-slave-smpn krb5kdc[23161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707529, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:45:29.995760 23822 tablet_bootstrap.cc:492] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:45:29.996024 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:29.980267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:34415)
0929 18:45:29.980771 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:29.980813 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:29.981646 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:29.982780 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:29.982788 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:29.983265 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:45:29.983891 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:29.983915 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:29.985706 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:29.985709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:29.986597 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:29.986606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:29.987374 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:29.987378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:29.987563 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:29.988111 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:45:29.988127 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:45:29.989912 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:45:29.992427 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:29.992433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:29.992448 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:29.992801 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:29.993221 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:29.993224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:29.993226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:29.993408 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:29.994104 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:45:29.994110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:45:29.994426 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 18:45:29.994659 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 18:45:29.995025 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":145,"threads_started":1}
I20260929 18:45:29.996200 23822 tablet_bootstrap.cc:492] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: Bootstrap complete.
I20260929 18:45:29.997262 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:29.980185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:34415 (local address 127.7.157.126:41607)
0929 18:45:29.980539 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:29.980561 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:29.980936 (+ 375us) server_negotiation.cc:407] Connection header received
0929 18:45:29.982061 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:29.982072 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:29.982380 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:29.982702 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:45:29.984114 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:29.985535 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:29.986718 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:29.987241 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:29.987392 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:29.990146 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:45:29.990152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:45:29.990170 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:45:29.990206 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:45:29.992293 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:29.992936 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:29.992943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:29.992949 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:29.993118 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:29.993516 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:29.993519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:29.993521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:29.993968 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:45:29.994109 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:29.995277 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0929 18:45:29.995579 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":101,"threads_started":1}
I20260929 18:45:29.997422 23822 ts_tablet_manager.cc:1397] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.007s
I20260929 18:45:29.999634 23930 heartbeater.cc:347] Connected to a master server at 127.7.157.126:41607
I20260929 18:45:29.999871 23930 heartbeater.cc:464] Registering TS with master...
I20260929 18:45:30.000432 23930 heartbeater.cc:511] Master 127.7.157.126:41607 requested a full tablet report, sending...
I20260929 18:45:30.000084 23822 raft_consensus.cc:348] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } } peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } }
I20260929 18:45:30.000589 23822 raft_consensus.cc:729] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a874820d3f425a81722abd8c246f2e, State: Initialized, Role: FOLLOWER
I20260929 18:45:30.001056 23822 consensus_queue.cc:260] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } } peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } }
I20260929 18:45:30.002045 23822 ts_tablet_manager.cc:1428] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:45:30.003826 23742 ts_manager.cc:194] Registered new tserver with Master: 74a874820d3f425a81722abd8c246f2e (127.7.157.65:40143)
I20260929 18:45:30.008203 23742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:34415
I20260929 18:45:30.010751 23930 heartbeater.cc:503] Master 127.7.157.126:41607 was elected leader, sending a full tablet report...
Sep 29 18:45:30 dist-test-slave-smpn krb5kdc[23161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707529, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for kudu/127.7.157.65@KRBTEST.COM
I20260929 18:45:30.017652 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.007224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.7.157.65:40143)
0929 18:45:30.007915 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:30.007923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:30.007941 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:45:30.007993 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:30.007997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:30.008149 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:30.008265 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:45:30.008989 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.009978 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.011174 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.011609 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.011710 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.014298 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:45:30.014307 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:45:30.014317 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:45:30.014339 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:45:30.015849 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:30.016321 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:30.016327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:30.016329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:30.016405 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:30.016669 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:30.016672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:30.016674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:30.016907 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:45:30.017033 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:30.017342 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 18:45:30.017504 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":95,"threads_started":1}
I20260929 18:45:30.017652 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.007150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:40143 (local address 127.0.0.1:55048)
0929 18:45:30.007536 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:30.007579 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:30.007702 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:30.008334 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:30.008342 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:30.008754 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:45:30.008860 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:30.008867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.010106 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.010109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:30.011028 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:30.011041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.011720 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.011724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:30.011832 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.012568 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:45:30.012584 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:45:30.014161 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:45:30.015983 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:30.015993 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:30.015998 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:30.016214 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:30.016496 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:30.016499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:30.016502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:30.016579 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:30.017014 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:45:30.017019 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:45:30.017134 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 18:45:30.017250 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 18:45:30.017455 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":101,"threads_started":1}
I20260929 18:45:30.018656 23864 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7640db70534a67bf14fa5081109934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:45:09 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:55048
I20260929 18:45:30.019520 23939 tablet_replica.cc:330] stopping tablet replica
I20260929 18:45:30.019845 23939 raft_consensus.cc:2229] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:45:30.020124 23939 raft_consensus.cc:2258] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:45:30.021291 23939 ts_tablet_manager.cc:1905] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:45:30.025780 23939 ts_tablet_manager.cc:1918] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:45:30.025894 23939 log.cc:1196] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal/wals/3c7640db70534a67bf14fa5081109934
I20260929 18:45:30.026180 23939 ts_tablet_manager.cc:1939] T 3c7640db70534a67bf14fa5081109934 P 74a874820d3f425a81722abd8c246f2e: Deleting consensus metadata
I20260929 18:45:30.028565 23729 catalog_manager.cc:4841] TS 74a874820d3f425a81722abd8c246f2e (127.7.157.65:40143): tablet 3c7640db70534a67bf14fa5081109934 (table test-table [id=238be7f256ef4f219332068f71593198]) successfully deleted
W20260929 18:45:30.104324 23934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:45:30.104578 23934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:45:30.104626 23934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:45:30.108712 23934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:45:30.108779 23934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:45:30.108841 23934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:45:30.114648 23934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:46149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=39375
--tserver_master_addrs=127.7.157.126:41607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:45:30.115857 23934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:45:30.116958 23934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:45:30.127049 23947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:30.127053 23944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:30.127053 23945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:30.127153 23934 server_base.cc:1034] running on GCE node
I20260929 18:45:30.127657 23934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:45:30.128348 23934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:45:30.129524 23934 hybrid_clock.cc:648] HybridClock initialized: now 1790707530129498 us; error 48 us; skew 500 ppm
Sep 29 18:45:30 dist-test-slave-smpn krb5kdc[23161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707530, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:45:30.132273 23934 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:45:30.133184 23934 webserver.cc:460] Webserver started at http://127.7.157.66:39375/ using document root <none> and password file <none>
I20260929 18:45:30.133746 23934 fs_manager.cc:359] Metadata directory not provided
I20260929 18:45:30.133833 23934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:45:30.137991 23934 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:45:30.140173 23954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:30.141444 23934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:45:30.141568 23934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "446764e8d4d44e90b7744b486190fec0"
format_stamp: "Formatted at 2026-09-29 18:45:09 on dist-test-slave-smpn"
server_key: "4de2ee2c0e0c294b4c2f2c34bb213439"
server_key_iv: "5be3bd01891026995776cf0bb44bef22"
server_key_version: "encryptionkey@0"
I20260929 18:45:30.142129 23934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:45:30.159147 23934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:45:30.162212 23934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:45:30.162389 23934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:45:30.162950 23934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:45:30.163806 23934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:45:30.163856 23934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:30.163908 23934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:45:30.163936 23934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:30.171972 23934 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:46149
I20260929 18:45:30.171998 24067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:46149 every 8 connection(s)
I20260929 18:45:30.173023 23934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:45:30.181350 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 23934
I20260929 18:45:30.181823 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:35489
--local_ip_for_outbound_sockets=127.7.157.67
--tserver_master_addrs=127.7.157.126:41607
--webserver_port=45739
--webserver_interface=127.7.157.67
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:45:30 dist-test-slave-smpn krb5kdc[23161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707530, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:45:30.187192 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.174850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:35915 (local address 127.7.157.126:41607)
0929 18:45:30.175033 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:30.175038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:30.175953 (+ 915us) server_negotiation.cc:407] Connection header received
0929 18:45:30.177203 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:30.177206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:30.177253 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:30.177326 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:45:30.178519 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.179191 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.180522 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.180765 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.180867 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.183602 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:45:30.183607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:45:30.183609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:45:30.183641 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:45:30.185131 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:30.185601 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:30.185604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:30.185606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:30.185701 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:30.186002 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:30.186005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:30.186006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:30.186234 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:45:30.186336 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:30.186818 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 18:45:30.186986 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:45:30.187894 24070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.175331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:35915)
0929 18:45:30.175796 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:30.175839 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:30.177007 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:30.177522 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:30.177529 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:30.177893 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:45:30.178383 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:30.178397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.179377 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.179380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:30.180388 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:30.180398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.180924 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.180927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:30.181109 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.181762 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:45:30.181782 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:45:30.183373 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:45:30.185241 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:30.185246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:30.185259 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:30.185506 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:30.185785 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:30.185787 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:30.185789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:30.185919 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:30.186322 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:45:30.186327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:45:30.186568 (+ 241us) client_negotiation.cc:769] Sending connection context
0929 18:45:30.186722 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 18:45:30.187001 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":121,"threads_started":1}
I20260929 18:45:30.189069 24068 heartbeater.cc:347] Connected to a master server at 127.7.157.126:41607
I20260929 18:45:30.189301 24068 heartbeater.cc:464] Registering TS with master...
I20260929 18:45:30.189776 24068 heartbeater.cc:511] Master 127.7.157.126:41607 requested a full tablet report, sending...
I20260929 18:45:30.190784 23742 ts_manager.cc:194] Registered new tserver with Master: 446764e8d4d44e90b7744b486190fec0 (127.7.157.66:46149)
I20260929 18:45:30.191371 23742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:35915
W20260929 18:45:30.292503 24071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:45:30.292769 24071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:45:30.292829 24071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:45:30.297073 24071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:45:30.297139 24071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:45:30.297222 24071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:45:30.303794 24071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:35489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=45739
--tserver_master_addrs=127.7.157.126:41607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:45:30.304885 24071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:45:30.305954 24071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:45:30.315560 24077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:30.315552 24076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:30.315657 24079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:30.315892 24071 server_base.cc:1034] running on GCE node
I20260929 18:45:30.316282 24071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:45:30.316901 24071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:45:30.318090 24071 hybrid_clock.cc:648] HybridClock initialized: now 1790707530318058 us; error 41 us; skew 500 ppm
Sep 29 18:45:30 dist-test-slave-smpn krb5kdc[23161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707530, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:45:30.320806 24071 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:45:30.321677 24071 webserver.cc:460] Webserver started at http://127.7.157.67:45739/ using document root <none> and password file <none>
I20260929 18:45:30.322240 24071 fs_manager.cc:359] Metadata directory not provided
I20260929 18:45:30.322309 24071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:45:30.326025 24071 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:45:30.327999 24086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:30.329202 24071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:45:30.329312 24071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "1567cd8458dc4a1db7f421c9f11417d7"
format_stamp: "Formatted at 2026-09-29 18:45:09 on dist-test-slave-smpn"
server_key: "85334c29ea1a6e4b2f7936017497446d"
server_key_iv: "6a38a04ce638ac398f4a6f5c40d920e7"
server_key_version: "encryptionkey@0"
I20260929 18:45:30.329816 24071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:45:30.346599 24071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:45:30.349814 24071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:45:30.350015 24071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:45:30.350589 24071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:45:30.351446 24071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:45:30.351493 24071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:30.351536 24071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:45:30.351564 24071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:30.359804 24071 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:35489
I20260929 18:45:30.359827 24199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:35489 every 8 connection(s)
I20260929 18:45:30.360783 24071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:45:30.368526 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 24071
Sep 29 18:45:30 dist-test-slave-smpn krb5kdc[23161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707530, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:45:30.374233 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.362553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:41969 (local address 127.7.157.126:41607)
0929 18:45:30.362686 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:30.362690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:30.363584 (+ 894us) server_negotiation.cc:407] Connection header received
0929 18:45:30.364809 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:30.364812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:30.364867 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:30.364949 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:45:30.366095 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.366607 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.367745 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.367973 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.368060 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.370679 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:45:30.370683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:45:30.370685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:45:30.370711 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:45:30.372122 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:30.372606 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:30.372609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:30.372610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:30.372697 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:30.373051 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:30.373055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:30.373057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:30.373241 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:45:30.373315 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:30.373949 (+ 634us) server_negotiation.cc:299] Negotiation successful
0929 18:45:30.374073 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:45:30.375191 24202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.362933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:41969)
0929 18:45:30.363403 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:30.363487 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:30.364608 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:30.365132 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:30.365138 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:30.365495 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:45:30.365964 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:30.365977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.366717 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.366720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:30.367637 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:30.367646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.368107 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.368111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:30.368284 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.368843 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:45:30.368859 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:45:30.370538 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:45:30.372231 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:30.372237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:30.372255 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:30.372522 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:30.372780 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:30.372783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:30.372785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:30.372958 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:30.373342 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:45:30.373348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:45:30.373693 (+ 345us) client_negotiation.cc:769] Sending connection context
0929 18:45:30.373898 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 18:45:30.374179 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":139,"threads_started":1}
I20260929 18:45:30.376313 24200 heartbeater.cc:347] Connected to a master server at 127.7.157.126:41607
I20260929 18:45:30.376574 24200 heartbeater.cc:464] Registering TS with master...
I20260929 18:45:30.377049 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.369187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.7.157.126:41607)
0929 18:45:30.369399 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:30.369403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:30.369561 (+ 158us) server_negotiation.cc:407] Connection header received
0929 18:45:30.369690 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:30.369693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:30.369733 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:30.369802 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:45:30.370629 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.371350 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.372289 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.372518 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.372602 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.373914 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:45:30.373917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:45:30.373919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:45:30.373959 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:45:30.375399 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:30.375851 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:30.375855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:30.375856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:30.375947 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:30.376217 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:30.376219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:30.376221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:30.376435 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:45:30.376652 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:30.376781 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 18:45:30.376896 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":65,"threads_started":1}
I20260929 18:45:30.377097 24200 heartbeater.cc:511] Master 127.7.157.126:41607 requested a full tablet report, sending...
I20260929 18:45:30.378208 23742 ts_manager.cc:194] Registered new tserver with Master: 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489)
I20260929 18:45:30.378834 23742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:41969
I20260929 18:45:30.390380 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:45:30.394038 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.390946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.7.157.126:41607)
0929 18:45:30.391091 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:30.391096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:30.391243 (+ 147us) server_negotiation.cc:407] Connection header received
0929 18:45:30.391401 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:30.391403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:30.391525 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:30.391605 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:45:30.391792 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.392298 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.393066 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.393283 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.393350 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.393581 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:45:30.393718 (+ 137us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 18:45:30.393922 (+ 204us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47868: token expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 18:45:30.394120 24203 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47868: token expired
W20260929 18:45:30.394783 23670 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.157.126:41607)
I20260929 18:45:30.402067 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.395183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.7.157.126:41607)
0929 18:45:30.395319 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:30.395323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:30.395336 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:45:30.395457 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:30.395460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:30.395512 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:30.395594 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:45:30.396196 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.396938 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.397684 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.398004 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.398091 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.399281 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:45:30.399284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:45:30.399287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:45:30.399307 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:45:30.400780 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:30.401149 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:30.401152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:30.401153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:30.401224 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:30.401407 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:30.401409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:30.401410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:30.401580 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:45:30.401674 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:30.401822 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 18:45:30.401909 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:45:30.404533 23742 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:45:30.407909 23742 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:45:30.415585 23864 tablet_service.cc:1467] Processing CreateTablet for tablet 7d8c5be606834a00a2adbe880904e1b4 (DEFAULT_TABLE table=test-table [id=112cb184e3f74147b00ae0f15e0f2e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:45:30.416185 23864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d8c5be606834a00a2adbe880904e1b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:45:30.418912 23822 tablet_bootstrap.cc:492] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e: Bootstrap starting.
I20260929 18:45:30.419842 23822 tablet_bootstrap.cc:654] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e: Neither blocks nor log segments found. Creating new log.
I20260929 18:45:30.420194 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.414849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:35489 (local address 127.0.0.1:48042)
0929 18:45:30.415148 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:30.415160 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:30.415240 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:30.416129 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:30.416132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:30.416151 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:45:30.416258 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:30.416265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.417413 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.417417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:30.418612 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:30.418624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.419735 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.419738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:30.419856 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.419892 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 18:45:30.419958 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 18:45:30.420033 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":12,"thread_start_us":96,"threads_started":1}
I20260929 18:45:30.420513 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.415323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48042 (local address 127.7.157.67:35489)
0929 18:45:30.415626 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:30.415638 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:30.415658 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:45:30.415722 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:30.415727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:30.415988 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:30.416116 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:45:30.416363 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.417306 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.418739 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.419633 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.419734 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.419989 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:30.420049 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 18:45:30.420145 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":90,"threads_started":1}
I20260929 18:45:30.422163 23822 tablet_bootstrap.cc:492] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e: No bootstrap required, opened a new log
I20260929 18:45:30.422253 23822 ts_tablet_manager.cc:1397] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:45:30.422730 23822 raft_consensus.cc:348] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } }
I20260929 18:45:30.422856 23822 raft_consensus.cc:374] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:45:30.422892 23822 raft_consensus.cc:729] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a874820d3f425a81722abd8c246f2e, State: Initialized, Role: FOLLOWER
I20260929 18:45:30.423018 23822 consensus_queue.cc:260] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } }
I20260929 18:45:30.423322 23822 ts_tablet_manager.cc:1428] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:45:30.423877 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.414567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:46149 (local address 127.0.0.1:57566)
0929 18:45:30.414671 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:30.414685 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:30.414785 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:30.419719 (+ 4934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:30.419722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:30.419742 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:45:30.419834 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:30.419840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.421374 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.421377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:30.422334 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:30.422345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.423453 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.423457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:30.423554 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.423580 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:45:30.423659 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:45:30.423738 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 18:45:30.423928 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.415325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.7.157.66:46149)
0929 18:45:30.419199 (+ 3874us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:30.419213 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:30.419239 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:45:30.419317 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:30.419323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:30.419552 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:30.420012 (+ 460us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:45:30.420201 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.421242 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.422486 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.423320 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.423428 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.423568 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:30.423700 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 18:45:30.423786 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3765,"thread_start_us":92,"threads_started":1}
I20260929 18:45:30.423493 24134 tablet_service.cc:1467] Processing CreateTablet for tablet 7d8c5be606834a00a2adbe880904e1b4 (DEFAULT_TABLE table=test-table [id=112cb184e3f74147b00ae0f15e0f2e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:45:30.424612 24134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d8c5be606834a00a2adbe880904e1b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:45:30.426232 24002 tablet_service.cc:1467] Processing CreateTablet for tablet 7d8c5be606834a00a2adbe880904e1b4 (DEFAULT_TABLE table=test-table [id=112cb184e3f74147b00ae0f15e0f2e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:45:30.427129 24002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d8c5be606834a00a2adbe880904e1b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:45:30.432036 24213 tablet_bootstrap.cc:492] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7: Bootstrap starting.
I20260929 18:45:30.434401 24214 tablet_bootstrap.cc:492] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0: Bootstrap starting.
I20260929 18:45:30.434947 24213 tablet_bootstrap.cc:654] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7: Neither blocks nor log segments found. Creating new log.
I20260929 18:45:30.435981 24213 log.cc:824] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:45:30.436762 24214 tablet_bootstrap.cc:654] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0: Neither blocks nor log segments found. Creating new log.
I20260929 18:45:30.437520 24214 log.cc:824] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:45:30.437745 24213 tablet_bootstrap.cc:492] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7: No bootstrap required, opened a new log
I20260929 18:45:30.437917 24213 ts_tablet_manager.cc:1397] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 18:45:30.439402 24214 tablet_bootstrap.cc:492] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0: No bootstrap required, opened a new log
I20260929 18:45:30.439600 24214 ts_tablet_manager.cc:1397] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:45:30.440999 24213 raft_consensus.cc:348] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } }
I20260929 18:45:30.441251 24213 raft_consensus.cc:374] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:45:30.441310 24213 raft_consensus.cc:729] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1567cd8458dc4a1db7f421c9f11417d7, State: Initialized, Role: FOLLOWER
I20260929 18:45:30.441702 24213 consensus_queue.cc:260] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } }
I20260929 18:45:30.442351 24200 heartbeater.cc:503] Master 127.7.157.126:41607 was elected leader, sending a full tablet report...
I20260929 18:45:30.442637 24213 ts_tablet_manager.cc:1428] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260929 18:45:30.442727 24214 raft_consensus.cc:348] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } }
I20260929 18:45:30.442970 24214 raft_consensus.cc:374] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:45:30.443027 24214 raft_consensus.cc:729] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446764e8d4d44e90b7744b486190fec0, State: Initialized, Role: FOLLOWER
I20260929 18:45:30.443517 23936 raft_consensus.cc:482] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:45:30.443717 23936 raft_consensus.cc:504] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } }
I20260929 18:45:30.444792 23936 leader_election.cc:288] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 446764e8d4d44e90b7744b486190fec0 (127.7.157.66:46149), 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489)
I20260929 18:45:30.444944 24214 consensus_queue.cc:260] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } }
I20260929 18:45:30.446048 24214 ts_tablet_manager.cc:1428] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260929 18:45:30.446382 24068 heartbeater.cc:503] Master 127.7.157.126:41607 was elected leader, sending a full tablet report...
I20260929 18:45:30.449213 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.444991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:46149 (local address 127.7.157.65:54931)
0929 18:45:30.445108 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:30.445124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:30.445216 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:30.445847 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:30.445849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:30.445866 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:45:30.445993 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:30.446000 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.446898 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.446901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:30.448015 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:30.448026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.448786 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.448790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:30.448899 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.448949 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 18:45:30.449016 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:45:30.449088 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 18:45:30.449496 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.445394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:54931 (local address 127.7.157.66:46149)
0929 18:45:30.445655 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:30.445660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:30.445674 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:45:30.445716 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:30.445719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:30.445763 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:30.445832 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:45:30.446113 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.446793 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.448140 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.448682 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.448786 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.448838 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:30.449274 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 18:45:30.449361 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260929 18:45:30.450091 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.445536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:40243 (local address 127.7.157.67:35489)
0929 18:45:30.446007 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:30.446011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:30.446023 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:45:30.446059 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:30.446062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:30.446101 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:30.446166 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:45:30.446613 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.447344 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.448757 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.449302 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.449399 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.449442 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:30.449902 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 18:45:30.449979 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394}
I20260929 18:45:30.450145 24022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8c5be606834a00a2adbe880904e1b4" candidate_uuid: "74a874820d3f425a81722abd8c246f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446764e8d4d44e90b7744b486190fec0" is_pre_election: true
I20260929 18:45:30.450429 24022 raft_consensus.cc:2454] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a874820d3f425a81722abd8c246f2e in term 0.
I20260929 18:45:30.450616 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.445036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:35489 (local address 127.7.157.65:40243)
0929 18:45:30.445615 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:30.445630 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:30.445723 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:30.446198 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:30.446201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:30.446218 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:45:30.446480 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:30.446490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.447473 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.447476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:30.448620 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:30.448631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.449652 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.449655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:30.449765 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.449790 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:45:30.450254 (+ 464us) client_negotiation.cc:240] Negotiation successful
0929 18:45:30.450451 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":82,"threads_started":1}
I20260929 18:45:30.450948 23819 leader_election.cc:302] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 446764e8d4d44e90b7744b486190fec0, 74a874820d3f425a81722abd8c246f2e; no voters:
I20260929 18:45:30.451288 23936 raft_consensus.cc:2781] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:45:30.451331 24154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8c5be606834a00a2adbe880904e1b4" candidate_uuid: "74a874820d3f425a81722abd8c246f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1567cd8458dc4a1db7f421c9f11417d7" is_pre_election: true
I20260929 18:45:30.451444 23936 raft_consensus.cc:482] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:45:30.451505 23936 raft_consensus.cc:3037] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:45:30.451606 24154 raft_consensus.cc:2454] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a874820d3f425a81722abd8c246f2e in term 0.
I20260929 18:45:30.452772 23936 raft_consensus.cc:504] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } }
I20260929 18:45:30.453146 23936 leader_election.cc:288] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 1 election: Requested vote from peers 446764e8d4d44e90b7744b486190fec0 (127.7.157.66:46149), 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489)
I20260929 18:45:30.453464 24154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8c5be606834a00a2adbe880904e1b4" candidate_uuid: "74a874820d3f425a81722abd8c246f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1567cd8458dc4a1db7f421c9f11417d7"
I20260929 18:45:30.453511 24022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8c5be606834a00a2adbe880904e1b4" candidate_uuid: "74a874820d3f425a81722abd8c246f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446764e8d4d44e90b7744b486190fec0"
I20260929 18:45:30.453568 24154 raft_consensus.cc:3037] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:45:30.453640 24022 raft_consensus.cc:3037] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:45:30.454558 24154 raft_consensus.cc:2454] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a874820d3f425a81722abd8c246f2e in term 1.
I20260929 18:45:30.454820 23818 leader_election.cc:302] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1567cd8458dc4a1db7f421c9f11417d7, 74a874820d3f425a81722abd8c246f2e; no voters:
I20260929 18:45:30.454880 24022 raft_consensus.cc:2454] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a874820d3f425a81722abd8c246f2e in term 1.
I20260929 18:45:30.454990 23936 raft_consensus.cc:2781] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:45:30.455231 23936 raft_consensus.cc:686] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 LEADER]: Becoming Leader. State: Replica: 74a874820d3f425a81722abd8c246f2e, State: Running, Role: LEADER
I20260929 18:45:30.455487 23936 consensus_queue.cc:237] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } }
I20260929 18:45:30.457821 23741 catalog_manager.cc:5495] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a874820d3f425a81722abd8c246f2e (127.7.157.65). New cstate: current_term: 1 leader_uuid: "74a874820d3f425a81722abd8c246f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } health_report { overall_health: HEALTHY } } }
I20260929 18:45:30.469106 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.465534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.7.157.65:40143)
0929 18:45:30.465644 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:30.465649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:30.465664 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:45:30.465843 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:30.465847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:30.465903 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:30.465998 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:45:30.466251 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.466786 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.467681 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:30.467932 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:30.468031 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:30.468296 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:45:30.468708 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:45:30.468827 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:30.468904 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 18:45:30.468963 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 18:45:30.472842 24022 raft_consensus.cc:1261] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0 [term 1 FOLLOWER]: Refusing update from remote peer 74a874820d3f425a81722abd8c246f2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:45:30.472842 24154 raft_consensus.cc:1261] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7 [term 1 FOLLOWER]: Refusing update from remote peer 74a874820d3f425a81722abd8c246f2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:45:30.473537 24220 consensus_queue.cc:1035] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:45:30.473708 23936 consensus_queue.cc:1035] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:45:30.479233 23931 tablet.cc:2369] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:45:30.489548 23742 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:45:30.489724 23742 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:45:30.492254 23742 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Sending DeleteTablet for 3 replicas of tablet 7d8c5be606834a00a2adbe880904e1b4
I20260929 18:45:30.493070 23864 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d8c5be606834a00a2adbe880904e1b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:45:30 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:55048
I20260929 18:45:30.493073 24134 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d8c5be606834a00a2adbe880904e1b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:45:30 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:48042
I20260929 18:45:30.493074 24002 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d8c5be606834a00a2adbe880904e1b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:45:30 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:57566
I20260929 18:45:30.493264 23939 tablet_replica.cc:330] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e: stopping tablet replica
I20260929 18:45:30.493383 23939 raft_consensus.cc:2229] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:45:30.493595 24232 tablet_replica.cc:330] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7: stopping tablet replica
I20260929 18:45:30.493734 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 23780
I20260929 18:45:30.493741 23939 raft_consensus.cc:2258] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:45:30.493923 24232 raft_consensus.cc:2229] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:45:30.494237 24232 raft_consensus.cc:2258] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:45:30.494493 24233 tablet_replica.cc:330] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0: stopping tablet replica
I20260929 18:45:30.494817 24233 raft_consensus.cc:2229] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:45:30.495208 24233 raft_consensus.cc:2258] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:45:30.496639 24233 ts_tablet_manager.cc:1905] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:45:30.497893 24232 ts_tablet_manager.cc:1905] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:45:30.498947 24233 ts_tablet_manager.cc:1918] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:45:30.499032 24233 log.cc:1196] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/7d8c5be606834a00a2adbe880904e1b4
I20260929 18:45:30.499341 24233 ts_tablet_manager.cc:1939] T 7d8c5be606834a00a2adbe880904e1b4 P 446764e8d4d44e90b7744b486190fec0: Deleting consensus metadata
I20260929 18:45:30.500695 24232 ts_tablet_manager.cc:1918] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260929 18:45:30.500681 23729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:40143 (error 108)
I20260929 18:45:30.500787 24232 log.cc:1196] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/7d8c5be606834a00a2adbe880904e1b4
I20260929 18:45:30.500782 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 23934
I20260929 18:45:30.501155 23729 catalog_manager.cc:4841] TS 446764e8d4d44e90b7744b486190fec0 (127.7.157.66:46149): tablet 7d8c5be606834a00a2adbe880904e1b4 (table test-table [id=112cb184e3f74147b00ae0f15e0f2e00]) successfully deleted
I20260929 18:45:30.501566 24232 ts_tablet_manager.cc:1939] T 7d8c5be606834a00a2adbe880904e1b4 P 1567cd8458dc4a1db7f421c9f11417d7: Deleting consensus metadata
I20260929 18:45:30.502626 23728 catalog_manager.cc:4841] TS 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489): tablet 7d8c5be606834a00a2adbe880904e1b4 (table test-table [id=112cb184e3f74147b00ae0f15e0f2e00]) successfully deleted
I20260929 18:45:30.504324 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:30.504047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:40143 (local address 127.0.0.1:55064)
0929 18:45:30.504174 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:30.504244 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:40143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 18:45:30.504483 23729 catalog_manager.cc:4584] TS 74a874820d3f425a81722abd8c246f2e (127.7.157.65:40143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d8c5be606834a00a2adbe880904e1b4: Network error: Client connection negotiation failed: client connection to 127.7.157.65:40143: connect: Connection refused (error 111)
I20260929 18:45:30.506104 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 24071
I20260929 18:45:30.510636 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 23709
I20260929 18:45:50.516194 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:41607
--webserver_interface=127.7.157.126
--webserver_port=42341
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:41607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:45:50.623764 24235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:45:50.624017 24235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:45:50.624069 24235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:45:50.627990 24235 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:45:50.628057 24235 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:45:50.628080 24235 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:45:50.628098 24235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:45:50.628120 24235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:45:50.628146 24235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:45:50.628176 24235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:45:50.628197 24235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:45:50.633325 24235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:41607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:41607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=42341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:45:50.634423 24235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:45:50.635478 24235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:45:50.643807 24241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:50.643807 24240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:50.643865 24235 server_base.cc:1034] running on GCE node
W20260929 18:45:50.643816 24243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:50.644603 24235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:45:50.645725 24235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:45:50.646884 24235 hybrid_clock.cc:648] HybridClock initialized: now 1790707550646864 us; error 30 us; skew 500 ppm
Sep 29 18:45:50 dist-test-slave-smpn krb5kdc[23161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707550, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:45:50.653379 24235 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:45:50.654356 24235 webserver.cc:460] Webserver started at http://127.7.157.126:42341/ using document root <none> and password file <none>
I20260929 18:45:50.654969 24235 fs_manager.cc:359] Metadata directory not provided
I20260929 18:45:50.655038 24235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:45:50.658846 24235 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:45:50.660941 24250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:50.662207 24235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:45:50.662320 24235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "999b21f5f102427a9127b14ed9d5bd4f"
format_stamp: "Formatted at 2026-09-29 18:45:08 on dist-test-slave-smpn"
server_key: "519648802149821875a642318975c2d8"
server_key_iv: "06320bd8a11792e1cfd67f72014d1f70"
server_key_version: "encryptionkey@0"
I20260929 18:45:50.662853 24235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:45:50.700062 24235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:45:50.703173 24235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:45:50.703349 24235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:45:50.710050 24235 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:41607
I20260929 18:45:50.710305 24302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:41607 every 8 connection(s)
I20260929 18:45:50.711131 24235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:45:50.712561 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 24235
I20260929 18:45:50.712947 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:40143
--local_ip_for_outbound_sockets=127.7.157.65
--tserver_master_addrs=127.7.157.126:41607
--webserver_port=34707
--webserver_interface=127.7.157.65
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:45:50.715392 24303 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:45:50.716996 24303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Bootstrap starting.
I20260929 18:45:50.733964 24303 log.cc:824] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:45:50.745100 24303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:45:50.745461 24303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Bootstrap complete.
I20260929 18:45:50.749275 24303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } }
I20260929 18:45:50.749655 24303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999b21f5f102427a9127b14ed9d5bd4f, State: Initialized, Role: FOLLOWER
I20260929 18:45:50.750099 24303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } }
I20260929 18:45:50.750226 24303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:45:50.750313 24303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:45:50.750430 24303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:45:50.752315 24303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } }
I20260929 18:45:50.752616 24303 leader_election.cc:302] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 999b21f5f102427a9127b14ed9d5bd4f; no voters:
I20260929 18:45:50.752909 24303 leader_election.cc:288] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 18:45:50.753074 24307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:45:50.753334 24307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [term 3 LEADER]: Becoming Leader. State: Replica: 999b21f5f102427a9127b14ed9d5bd4f, State: Running, Role: LEADER
I20260929 18:45:50.753669 24307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } }
I20260929 18:45:50.754190 24303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:45:50.755007 24308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "999b21f5f102427a9127b14ed9d5bd4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } } }
I20260929 18:45:50.755159 24308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [sys.catalog]: This master's current role is: LEADER
I20260929 18:45:50.755534 24309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 999b21f5f102427a9127b14ed9d5bd4f. Latest consensus state: current_term: 3 leader_uuid: "999b21f5f102427a9127b14ed9d5bd4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999b21f5f102427a9127b14ed9d5bd4f" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41607 } } }
I20260929 18:45:50.755662 24309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f [sys.catalog]: This master's current role is: LEADER
I20260929 18:45:50.755957 24316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:45:50.761036 24316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:45:50.761646 24316 catalog_manager.cc:1259] Loaded cluster ID: b70889f9c38a414b97993279027d9766
I20260929 18:45:50.761705 24316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:45:50.764654 24316 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:45:50.765413 24316 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Loaded TSK: 0
I20260929 18:45:50.765496 24316 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 999b21f5f102427a9127b14ed9d5bd4f: Loaded TSK: 1
F20260929 18:45:50.766319 24316 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc44a9729ad google::LogMessage::Fail() at ??:0
@ 0x7fc44a97588a google::LogMessage::SendToLog() at ??:0
@ 0x7fc44a9724de google::LogMessage::Flush() at ??:0
@ 0x7fc44a973a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc44d63618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc44d62ef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc4552d77bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc4552b199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc4552b2178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fc4552f3df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc45531282b std::function<>::operator()() at ??:0
@ 0x7fc4552b1b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc4552b2f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc4552af579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc4552f3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc44cbc43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc44cbc4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc44cbc63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc44cbb5512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc44e1f96db start_thread at ??:0
@ 0x7fc44947061f clone at ??:0
W20260929 18:45:50.854468 24305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:45:50.854878 24305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:45:50.854954 24305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:45:50.861083 24305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:45:50.861186 24305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:45:50.861289 24305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:45:50.869208 24305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:40143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=34707
--tserver_master_addrs=127.7.157.126:41607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:45:50.870640 24305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:45:50.872023 24305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:45:50.883301 24350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:50.883301 24351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:50.883415 24305 server_base.cc:1034] running on GCE node
W20260929 18:45:50.883339 24353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:50.884111 24305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:45:50.884950 24305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:45:50.886157 24305 hybrid_clock.cc:648] HybridClock initialized: now 1790707550886126 us; error 40 us; skew 500 ppm
Sep 29 18:45:50 dist-test-slave-smpn krb5kdc[23161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707550, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:45:50.889564 24305 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:45:50.890740 24305 webserver.cc:460] Webserver started at http://127.7.157.65:34707/ using document root <none> and password file <none>
I20260929 18:45:50.891499 24305 fs_manager.cc:359] Metadata directory not provided
I20260929 18:45:50.891588 24305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:45:50.896721 24305 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 18:45:50.899418 24360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:50.900846 24305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 18:45:50.900971 24305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "74a874820d3f425a81722abd8c246f2e"
format_stamp: "Formatted at 2026-09-29 18:45:08 on dist-test-slave-smpn"
server_key: "4037efa49659c2a62d778cc97877858c"
server_key_iv: "2e264b41d7c8fbd031bfe2c679542c1a"
server_key_version: "encryptionkey@0"
I20260929 18:45:50.901542 24305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:45:50.922199 24305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:45:50.925596 24305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:45:50.925804 24305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:45:50.926443 24305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:45:50.927711 24367 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:45:50.933331 24305 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:45:50.933419 24305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 18:45:50.933497 24305 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:45:50.935381 24305 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:45:50.935453 24305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260929 18:45:50.935771 24367 tablet_bootstrap.cc:492] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e: Bootstrap starting.
I20260929 18:45:50.946090 24305 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:40143
I20260929 18:45:50.946344 24474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:40143 every 8 connection(s)
I20260929 18:45:50.947160 24305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:45:50.949681 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 24305
I20260929 18:45:50.950117 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:46149
--local_ip_for_outbound_sockets=127.7.157.66
--tserver_master_addrs=127.7.157.126:41607
--webserver_port=39375
--webserver_interface=127.7.157.66
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:45:50.958015 24367 log.cc:824] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:45:50.967020 24367 tablet_bootstrap.cc:492] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:45:50.967458 24367 tablet_bootstrap.cc:492] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e: Bootstrap complete.
I20260929 18:45:50.968767 24367 ts_tablet_manager.cc:1397] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e: Time spent bootstrapping tablet: real 0.033s user 0.016s sys 0.014s
I20260929 18:45:50.972019 24367 raft_consensus.cc:348] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } }
I20260929 18:45:50.972559 24367 raft_consensus.cc:729] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a874820d3f425a81722abd8c246f2e, State: Initialized, Role: FOLLOWER
I20260929 18:45:50.973073 24367 consensus_queue.cc:260] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } }
I20260929 18:45:50.974123 24367 ts_tablet_manager.cc:1428] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260929 18:45:51.007313 24477 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:45:50.949033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:43291)
0929 18:45:50.950083 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:50.950142 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:50.951359 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:51.006485 (+ 55126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":112,"threads_started":1}
W20260929 18:45:51.007794 24475 heartbeater.cc:644] Failed to heartbeat to 127.7.157.126:41607 (0 consecutive failures): Network error: Failed to ping master at 127.7.157.126:41607: Client connection negotiation failed: client connection to 127.7.157.126:41607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:45:51.008670 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:51.008410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:53529)
0929 18:45:51.008539 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:51.008609 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 18:45:51.009343 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:51.009115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:55495)
0929 18:45:51.009249 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:51.009293 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 18:45:51.009532 24475 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:45:51.063979 24478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:45:51.064231 24478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:45:51.064272 24478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:45:51.068140 24478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:45:51.068189 24478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:45:51.068245 24478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:45:51.073283 24478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:46149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=39375
--tserver_master_addrs=127.7.157.126:41607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:45:51.074246 24478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:45:51.075234 24478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:45:51.084282 24485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:51.084229 24486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:51.084236 24488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:51.084314 24478 server_base.cc:1034] running on GCE node
I20260929 18:45:51.084834 24478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:45:51.085433 24478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:45:51.086598 24478 hybrid_clock.cc:648] HybridClock initialized: now 1790707551086578 us; error 36 us; skew 500 ppm
Sep 29 18:45:51 dist-test-slave-smpn krb5kdc[23161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707551, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:45:51.089103 24478 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:45:51.089902 24478 webserver.cc:460] Webserver started at http://127.7.157.66:39375/ using document root <none> and password file <none>
I20260929 18:45:51.090418 24478 fs_manager.cc:359] Metadata directory not provided
I20260929 18:45:51.090482 24478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:45:51.094229 24478 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:45:51.096263 24495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:51.097458 24478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:45:51.097564 24478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "446764e8d4d44e90b7744b486190fec0"
format_stamp: "Formatted at 2026-09-29 18:45:09 on dist-test-slave-smpn"
server_key: "4de2ee2c0e0c294b4c2f2c34bb213439"
server_key_iv: "5be3bd01891026995776cf0bb44bef22"
server_key_version: "encryptionkey@0"
I20260929 18:45:51.098105 24478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:45:51.131018 24478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:45:51.133987 24478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:45:51.134181 24478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:45:51.134804 24478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:45:51.135645 24478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:45:51.135694 24478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:51.135756 24478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:45:51.135787 24478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:51.144302 24478 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:46149
I20260929 18:45:51.144331 24608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:46149 every 8 connection(s)
I20260929 18:45:51.145418 24478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:45:51.147626 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 24478
I20260929 18:45:51.147994 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:35489
--local_ip_for_outbound_sockets=127.7.157.67
--tserver_master_addrs=127.7.157.126:41607
--webserver_port=45739
--webserver_interface=127.7.157.67
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:45:51.149438 24611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:51.147551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:38033)
0929 18:45:51.148022 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:51.148520 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":124,"threads_started":1}
W20260929 18:45:51.149895 24609 heartbeater.cc:644] Failed to heartbeat to 127.7.157.126:41607 (0 consecutive failures): Network error: Failed to ping master at 127.7.157.126:41607: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
I20260929 18:45:51.150484 24611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:51.150251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:32953)
0929 18:45:51.150370 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:51.150423 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 18:45:51.151206 24611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:51.150861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:43991)
0929 18:45:51.151100 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:51.151148 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260929 18:45:51.151364 24609 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:45:51.255986 24612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:45:51.256258 24612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:45:51.256302 24612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:45:51.260311 24612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:45:51.260371 24612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:45:51.260428 24612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:45:51.265561 24612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:35489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=45739
--tserver_master_addrs=127.7.157.126:41607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:45:51.266523 24612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:45:51.267508 24612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:45:51.276516 24620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:51.276523 24617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:45:51.276523 24618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:45:51.276721 24612 server_base.cc:1034] running on GCE node
I20260929 18:45:51.277151 24612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:45:51.277796 24612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:45:51.278957 24612 hybrid_clock.cc:648] HybridClock initialized: now 1790707551278942 us; error 29 us; skew 500 ppm
Sep 29 18:45:51 dist-test-slave-smpn krb5kdc[23161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707551, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:45:51.281879 24612 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:45:51.282770 24612 webserver.cc:460] Webserver started at http://127.7.157.67:45739/ using document root <none> and password file <none>
I20260929 18:45:51.283370 24612 fs_manager.cc:359] Metadata directory not provided
I20260929 18:45:51.283438 24612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:45:51.287341 24612 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:45:51.289387 24627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:51.290612 24612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:45:51.290735 24612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "1567cd8458dc4a1db7f421c9f11417d7"
format_stamp: "Formatted at 2026-09-29 18:45:09 on dist-test-slave-smpn"
server_key: "85334c29ea1a6e4b2f7936017497446d"
server_key_iv: "6a38a04ce638ac398f4a6f5c40d920e7"
server_key_version: "encryptionkey@0"
I20260929 18:45:51.291312 24612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:45:51.329422 24612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:45:51.332577 24612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:45:51.332770 24612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:45:51.333400 24612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:45:51.334261 24612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:45:51.334311 24612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:51.334364 24612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:45:51.334389 24612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:45:51.342890 24612 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:35489
I20260929 18:45:51.342919 24740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:35489 every 8 connection(s)
I20260929 18:45:51.343953 24612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:45:51.344769 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 24612
I20260929 18:45:51.347815 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:51.346156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:43339)
0929 18:45:51.346616 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:51.347024 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":103,"threads_started":1}
W20260929 18:45:51.348232 24741 heartbeater.cc:644] Failed to heartbeat to 127.7.157.126:41607 (0 consecutive failures): Network error: Failed to ping master at 127.7.157.126:41607: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
I20260929 18:45:51.348806 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:51.348599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:38117)
0929 18:45:51.348703 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:51.348750 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 18:45:51.349373 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:51.349184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:36571)
0929 18:45:51.349284 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:51.349325 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:45:51.349529 24741 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:45:52.010599 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:52.010134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:47279)
0929 18:45:52.010433 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:52.010493 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":113,"threads_started":1}
I20260929 18:45:52.146442 24746 raft_consensus.cc:482] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:45:52.146639 24746 raft_consensus.cc:504] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } }
I20260929 18:45:52.147805 24746 leader_election.cc:288] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 446764e8d4d44e90b7744b486190fec0 (127.7.157.66:46149), 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489)
I20260929 18:45:52.152165 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:52.151809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:44029)
0929 18:45:52.152016 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:52.152063 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":76,"threads_started":1}
Sep 29 18:45:52 dist-test-slave-smpn krb5kdc[23161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707550, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.67@KRBTEST.COM
Sep 29 18:45:52 dist-test-slave-smpn krb5kdc[23161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707550, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.66@KRBTEST.COM
I20260929 18:45:52.163512 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:52.148004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:46149 (local address 127.7.157.65:43095)
0929 18:45:52.148188 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:52.148216 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:52.148315 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:52.150170 (+ 1855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:52.150176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:52.150749 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:45:52.151601 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:52.151623 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:52.153488 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:52.153491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:52.154858 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:52.154869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:52.155830 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:52.155833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:52.156100 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:52.156753 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:45:52.156774 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:45:52.158948 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:45:52.161011 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:52.161016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:52.161027 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:52.161275 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:52.161578 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:52.161581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:52.161583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:52.161730 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:52.162512 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:45:52.162519 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:45:52.162773 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 18:45:52.162930 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 18:45:52.163313 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":249,"spinlock_wait_cycles":54784}
I20260929 18:45:52.163522 24749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:52.148355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:55397 (local address 127.7.157.67:35489)
0929 18:45:52.148736 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:52.148747 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:52.148780 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:45:52.149503 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:52.149510 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:52.149821 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:52.150072 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:45:52.152063 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:52.153614 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:52.154993 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:52.155718 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:52.155879 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:52.158914 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:45:52.158919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:45:52.158932 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:45:52.158960 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:45:52.160890 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:52.161605 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:52.161610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:52.161613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:52.161753 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:52.162047 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:52.162051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:52.162052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:52.162507 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:45:52.162628 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:52.163062 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 18:45:52.163364 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":83,"threads_started":1}
I20260929 18:45:52.163558 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:52.148077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:35489 (local address 127.7.157.65:55397)
0929 18:45:52.148500 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:52.148510 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:45:52.148581 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:45:52.150170 (+ 1589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:45:52.150176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:45:52.150750 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:45:52.151608 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:52.151623 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:52.153757 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:52.153761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:52.154861 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:45:52.154869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:45:52.155830 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:45:52.155833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:45:52.156097 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:52.156794 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:45:52.156806 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:45:52.158703 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:45:52.161359 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:52.161360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:52.161362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:52.161506 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:52.161862 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:45:52.161865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:45:52.161867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:45:52.161959 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:45:52.162608 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:45:52.162610 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:45:52.162770 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 18:45:52.162930 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 18:45:52.163396 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":363,"spinlock_wait_cycles":4736,"thread_start_us":65,"threads_started":1}
I20260929 18:45:52.163532 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:52.148117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:43095 (local address 127.7.157.66:46149)
0929 18:45:52.148420 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 18:45:52.148435 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:45:52.148478 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:45:52.149313 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:45:52.149321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:45:52.149747 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:45:52.150072 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:45:52.151777 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:52.153349 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:52.154975 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:45:52.155715 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:45:52.155879 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:45:52.159147 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:45:52.159153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:45:52.159170 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:45:52.159209 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:45:52.160890 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:52.161366 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:52.161370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:52.161374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:52.161479 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:45:52.161828 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:45:52.161831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:45:52.161832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:45:52.162325 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:45:52.162483 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 18:45:52.163088 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 18:45:52.163350 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":107,"threads_started":1}
W20260929 18:45:52.166368 24363 leader_election.cc:341] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489): Not found: Tablet not found: 7d8c5be606834a00a2adbe880904e1b4
W20260929 18:45:52.167289 24364 leader_election.cc:341] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 446764e8d4d44e90b7744b486190fec0 (127.7.157.66:46149): Not found: Tablet not found: 7d8c5be606834a00a2adbe880904e1b4
I20260929 18:45:52.167390 24364 leader_election.cc:302] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a874820d3f425a81722abd8c246f2e; no voters: 1567cd8458dc4a1db7f421c9f11417d7, 446764e8d4d44e90b7744b486190fec0
I20260929 18:45:52.167665 24746 raft_consensus.cc:2726] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:45:52.350458 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:52.350068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:36059)
0929 18:45:52.350293 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:52.350355 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260929 18:45:53.011819 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:53.011429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:49563)
0929 18:45:53.011670 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:53.011724 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260929 18:45:53.153162 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:53.152779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:43123)
0929 18:45:53.153018 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:53.153073 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":79,"threads_started":1}
I20260929 18:45:53.351650 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:53.351234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:35117)
0929 18:45:53.351482 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:53.351540 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260929 18:45:54.012957 24755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:54.012515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:35421)
0929 18:45:54.012799 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:54.012860 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":75,"threads_started":1}
I20260929 18:45:54.096676 24756 raft_consensus.cc:482] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:45:54.096810 24756 raft_consensus.cc:504] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } }
I20260929 18:45:54.097235 24756 leader_election.cc:288] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 446764e8d4d44e90b7744b486190fec0 (127.7.157.66:46149), 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489)
W20260929 18:45:54.098008 24364 leader_election.cc:341] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 446764e8d4d44e90b7744b486190fec0 (127.7.157.66:46149): Not found: Tablet not found: 7d8c5be606834a00a2adbe880904e1b4
W20260929 18:45:54.098138 24363 leader_election.cc:341] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489): Not found: Tablet not found: 7d8c5be606834a00a2adbe880904e1b4
I20260929 18:45:54.098237 24363 leader_election.cc:302] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a874820d3f425a81722abd8c246f2e; no voters: 1567cd8458dc4a1db7f421c9f11417d7, 446764e8d4d44e90b7744b486190fec0
I20260929 18:45:54.098385 24756 raft_consensus.cc:2726] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:45:54.154327 24757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:54.153916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:40647)
0929 18:45:54.154144 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:54.154218 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
I20260929 18:45:54.352720 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:54.352335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:50567)
0929 18:45:54.352567 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:54.352622 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260929 18:45:55.014128 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:55.013699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:45159)
0929 18:45:55.013971 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:55.014032 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260929 18:45:55.155586 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:55.155108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:55647)
0929 18:45:55.155371 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:55.155450 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":63,"threads_started":1}
I20260929 18:45:55.353827 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:55.353439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:51723)
0929 18:45:55.353677 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:55.353732 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260929 18:45:56.015597 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:56.015120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:57829)
0929 18:45:56.015425 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:56.015488 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":79,"threads_started":1}
I20260929 18:45:56.156801 24763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:56.156370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:45397)
0929 18:45:56.156628 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:56.156694 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260929 18:45:56.354912 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:56.354520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:60869)
0929 18:45:56.354765 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:56.354820 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":70,"threads_started":1}
I20260929 18:45:57.016697 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:57.016276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:60273)
0929 18:45:57.016518 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:57.016575 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":87,"threads_started":1}
I20260929 18:45:57.157944 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:57.157545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:41247)
0929 18:45:57.157791 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:57.157849 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":72,"threads_started":1}
I20260929 18:45:57.355980 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:57.355597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:59653)
0929 18:45:57.355828 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:57.355886 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260929 18:45:57.389535 24768 raft_consensus.cc:482] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:45:57.389705 24768 raft_consensus.cc:504] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } }
I20260929 18:45:57.390110 24768 leader_election.cc:288] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 446764e8d4d44e90b7744b486190fec0 (127.7.157.66:46149), 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489)
W20260929 18:45:57.390887 24364 leader_election.cc:341] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 446764e8d4d44e90b7744b486190fec0 (127.7.157.66:46149): Not found: Tablet not found: 7d8c5be606834a00a2adbe880904e1b4
W20260929 18:45:57.391026 24363 leader_election.cc:341] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489): Not found: Tablet not found: 7d8c5be606834a00a2adbe880904e1b4
I20260929 18:45:57.391162 24363 leader_election.cc:302] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a874820d3f425a81722abd8c246f2e; no voters: 1567cd8458dc4a1db7f421c9f11417d7, 446764e8d4d44e90b7744b486190fec0
I20260929 18:45:57.391310 24768 raft_consensus.cc:2726] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:45:58.017848 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:58.017436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:42891)
0929 18:45:58.017691 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:58.017753 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
I20260929 18:45:58.159188 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:58.158685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:48245)
0929 18:45:58.158973 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:58.159032 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":68,"threads_started":1}
I20260929 18:45:58.357365 24771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:58.356817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:56097)
0929 18:45:58.357193 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:58.357263 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":114,"threads_started":1}
I20260929 18:45:59.019023 24772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:59.018608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:50701)
0929 18:45:59.018867 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:59.018928 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":92,"threads_started":1}
I20260929 18:45:59.160374 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:59.159959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:53013)
0929 18:45:59.160218 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:59.160278 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260929 18:45:59.358557 24774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:45:59.358148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:50623)
0929 18:45:59.358394 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 18:45:59.358459 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
I20260929 18:46:00.020129 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:00.019723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:46947)
0929 18:46:00.019971 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:00.020031 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260929 18:46:00.161623 24776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:00.161148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:48843)
0929 18:46:00.161457 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:00.161520 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":95,"threads_started":1}
I20260929 18:46:00.359676 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:00.359289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:53469)
0929 18:46:00.359521 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:00.359580 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260929 18:46:01.021384 24778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:01.020947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:54641)
0929 18:46:01.021234 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:01.021287 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":73,"threads_started":1}
I20260929 18:46:01.162763 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:01.162360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:57201)
0929 18:46:01.162615 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:01.162668 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
I20260929 18:46:01.360841 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:01.360415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:44177)
0929 18:46:01.360682 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:01.360743 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20260929 18:46:01.470885 24781 raft_consensus.cc:482] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:46:01.471021 24781 raft_consensus.cc:504] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } }
I20260929 18:46:01.471443 24781 leader_election.cc:288] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 446764e8d4d44e90b7744b486190fec0 (127.7.157.66:46149), 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489)
W20260929 18:46:01.472180 24363 leader_election.cc:341] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489): Not found: Tablet not found: 7d8c5be606834a00a2adbe880904e1b4
W20260929 18:46:01.472297 24364 leader_election.cc:341] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 446764e8d4d44e90b7744b486190fec0 (127.7.157.66:46149): Not found: Tablet not found: 7d8c5be606834a00a2adbe880904e1b4
I20260929 18:46:01.472383 24364 leader_election.cc:302] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a874820d3f425a81722abd8c246f2e; no voters: 1567cd8458dc4a1db7f421c9f11417d7, 446764e8d4d44e90b7744b486190fec0
I20260929 18:46:01.472513 24781 raft_consensus.cc:2726] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:46:02.022527 24782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:02.022105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:53169)
0929 18:46:02.022374 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:02.022432 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":88,"threads_started":1}
I20260929 18:46:02.163874 24783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:02.163444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:55775)
0929 18:46:02.163720 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:02.163778 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":86,"threads_started":1}
I20260929 18:46:02.361989 24784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:02.361576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:48943)
0929 18:46:02.361832 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:02.361891 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260929 18:46:03.023671 24785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:03.023283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:37393)
0929 18:46:03.023517 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:03.023578 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
I20260929 18:46:03.165061 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:03.164625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:38601)
0929 18:46:03.164901 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:03.164962 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20260929 18:46:03.363389 24787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:03.362891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:46053)
0929 18:46:03.363208 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:03.363273 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
I20260929 18:46:04.024935 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:04.024502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:50393)
0929 18:46:04.024780 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:04.024840 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
I20260929 18:46:04.166285 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:04.165803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:40417)
0929 18:46:04.166098 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:04.166180 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
I20260929 18:46:04.364475 24790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:04.364076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:52255)
0929 18:46:04.364319 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:04.364377 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260929 18:46:05.026162 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:05.025736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:40789)
0929 18:46:05.026009 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:05.026067 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
I20260929 18:46:05.167616 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:05.167131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:46355)
0929 18:46:05.167446 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:05.167505 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":108,"threads_started":1}
I20260929 18:46:05.197810 24793 raft_consensus.cc:482] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:46:05.197961 24793 raft_consensus.cc:504] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446764e8d4d44e90b7744b486190fec0" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 46149 } } peers { permanent_uuid: "1567cd8458dc4a1db7f421c9f11417d7" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35489 } } peers { permanent_uuid: "74a874820d3f425a81722abd8c246f2e" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40143 } }
I20260929 18:46:05.198453 24793 leader_election.cc:288] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 446764e8d4d44e90b7744b486190fec0 (127.7.157.66:46149), 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489)
W20260929 18:46:05.199183 24364 leader_election.cc:341] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 446764e8d4d44e90b7744b486190fec0 (127.7.157.66:46149): Not found: Tablet not found: 7d8c5be606834a00a2adbe880904e1b4
W20260929 18:46:05.199402 24363 leader_election.cc:341] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1567cd8458dc4a1db7f421c9f11417d7 (127.7.157.67:35489): Not found: Tablet not found: 7d8c5be606834a00a2adbe880904e1b4
I20260929 18:46:05.199468 24363 leader_election.cc:302] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a874820d3f425a81722abd8c246f2e; no voters: 1567cd8458dc4a1db7f421c9f11417d7, 446764e8d4d44e90b7744b486190fec0
I20260929 18:46:05.199826 24793 raft_consensus.cc:2726] T 7d8c5be606834a00a2adbe880904e1b4 P 74a874820d3f425a81722abd8c246f2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:46:05.365653 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:05.365230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.67:58905)
0929 18:46:05.365475 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:05.365555 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260929 18:46:06.027436 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:06.027039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.65:55125)
0929 18:46:06.027292 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:06.027343 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
I20260929 18:46:06.169018 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:06.168646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41607 (local address 127.7.157.66:38963)
0929 18:46:06.168870 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:06.168922 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.126:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
I20260929 18:46:06.353116 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 24305
I20260929 18:46:06.358238 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 24478
I20260929 18:46:06.362782 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 24612
2026-09-29T18:46:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59028 ms)
[----------] 30 tests from SecurityITest (237736 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:06 dist-test-slave-smpn krb5kdc[24799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:06 dist-test-slave-smpn krb5kdc[24799](info): set up 2 sockets
Sep 29 18:46:06 dist-test-slave-smpn krb5kdc[24799](info): commencing operation
krb5kdc: starting...
W20260929 18:46:07.322642 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:07 dist-test-slave-smpn krb5kdc[24799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:46:07.437913 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:45531
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:43659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:45531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:07.546883 24812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:07.547148 24812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:07.547189 24812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:07.551097 24812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:07.551152 24812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:07.551170 24812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:07.551198 24812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:07.551224 24812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:07.556459 24812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:43659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:45531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:45531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:07.557467 24812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:07.558444 24812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:46:07.566718 24812 server_base.cc:1034] running on GCE node
W20260929 18:46:07.566876 24820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:07.566856 24817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:07.566895 24818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:07.567616 24812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:07.568667 24812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:07.569828 24812 hybrid_clock.cc:648] HybridClock initialized: now 1790707567569817 us; error 23 us; skew 500 ppm
Sep 29 18:46:07 dist-test-slave-smpn krb5kdc[24799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707567, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:07.572481 24812 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:46:07.573369 24812 webserver.cc:460] Webserver started at http://127.7.157.126:46135/ using document root <none> and password file <none>
I20260929 18:46:07.573892 24812 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:07.573956 24812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:07.574141 24812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:07.576160 24812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "be7573182f79400790e17f4ed2d0ffec"
format_stamp: "Formatted at 2026-09-29 18:46:07 on dist-test-slave-smpn"
server_key: "c7fc2744c79a31d7b7443d62d014a813"
server_key_iv: "bdea96f14b239b669b93040d9051e3d9"
server_key_version: "encryptionkey@0"
I20260929 18:46:07.576723 24812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "be7573182f79400790e17f4ed2d0ffec"
format_stamp: "Formatted at 2026-09-29 18:46:07 on dist-test-slave-smpn"
server_key: "c7fc2744c79a31d7b7443d62d014a813"
server_key_iv: "bdea96f14b239b669b93040d9051e3d9"
server_key_version: "encryptionkey@0"
I20260929 18:46:07.580268 24812 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:46:07.582612 24827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:07.583877 24812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:46:07.583981 24812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "be7573182f79400790e17f4ed2d0ffec"
format_stamp: "Formatted at 2026-09-29 18:46:07 on dist-test-slave-smpn"
server_key: "c7fc2744c79a31d7b7443d62d014a813"
server_key_iv: "bdea96f14b239b669b93040d9051e3d9"
server_key_version: "encryptionkey@0"
I20260929 18:46:07.584097 24812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:07.604923 24812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:07.608064 24812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:07.608237 24812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:07.615101 24812 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:45531
I20260929 18:46:07.615154 24879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:45531 every 8 connection(s)
I20260929 18:46:07.616074 24812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:07.618901 24880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:07.623739 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 24812
I20260929 18:46:07.623852 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:07.624114 7797 external_mini_cluster.cc:1428] Setting key edd60d6eedb01bfd9d6e1748fa3e8239
I20260929 18:46:07.624780 24880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec: Bootstrap starting.
I20260929 18:46:07.627399 24880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:07.628192 24880 log.cc:824] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:07.630245 24880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec: No bootstrap required, opened a new log
Sep 29 18:46:07 dist-test-slave-smpn krb5kdc[24799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:07.634306 24880 raft_consensus.cc:348] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7573182f79400790e17f4ed2d0ffec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45531 } }
I20260929 18:46:07.634564 24880 raft_consensus.cc:374] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:07.634627 24880 raft_consensus.cc:729] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be7573182f79400790e17f4ed2d0ffec, State: Initialized, Role: FOLLOWER
I20260929 18:46:07.635135 24880 consensus_queue.cc:260] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7573182f79400790e17f4ed2d0ffec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45531 } }
I20260929 18:46:07.635274 24880 raft_consensus.cc:388] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:07.635352 24880 raft_consensus.cc:482] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:07.635455 24880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:07.636363 24880 raft_consensus.cc:504] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7573182f79400790e17f4ed2d0ffec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45531 } }
I20260929 18:46:07.636691 24880 leader_election.cc:302] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be7573182f79400790e17f4ed2d0ffec; no voters:
I20260929 18:46:07.637003 24880 leader_election.cc:288] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:07.637099 24885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:07.637383 24885 raft_consensus.cc:686] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [term 1 LEADER]: Becoming Leader. State: Replica: be7573182f79400790e17f4ed2d0ffec, State: Running, Role: LEADER
I20260929 18:46:07.637902 24885 consensus_queue.cc:237] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7573182f79400790e17f4ed2d0ffec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45531 } }
I20260929 18:46:07.638352 24880 sys_catalog.cc:564] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:07.638759 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:07.625507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44172 (local address 127.7.157.126:45531)
0929 18:46:07.625976 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:07.625992 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:07.626030 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:46:07.626697 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:07.626715 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:07.627101 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:07.627438 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:07.628238 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:07.629449 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:07.630315 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:07.630849 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:07.630998 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:07.633708 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:07.633713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:07.633727 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:07.633754 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:07.635660 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:07.636069 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:07.636075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:07.636081 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:07.636196 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:07.636459 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:07.636462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:07.636464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:07.636929 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:07.637114 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:07.637463 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 18:46:07.637796 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":149,"threads_started":1}
I20260929 18:46:07.639118 24885 sys_catalog.cc:455] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [sys.catalog]: SysCatalogTable state changed. Reason: New leader be7573182f79400790e17f4ed2d0ffec. Latest consensus state: current_term: 1 leader_uuid: "be7573182f79400790e17f4ed2d0ffec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7573182f79400790e17f4ed2d0ffec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45531 } } }
I20260929 18:46:07.639338 24885 sys_catalog.cc:458] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:07.639613 24885 sys_catalog.cc:455] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be7573182f79400790e17f4ed2d0ffec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7573182f79400790e17f4ed2d0ffec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45531 } } }
I20260929 18:46:07.639750 24885 sys_catalog.cc:458] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:07.639779 24894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:07.642966 24894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:07.649206 24894 catalog_manager.cc:1347] Generated new cluster ID: 0de0dab12d1140d696e585c9313a9f9f
I20260929 18:46:07.649271 24894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:07.663805 24894 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:07.664814 24894 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:07.674263 24894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec: Generated new TSK 0
I20260929 18:46:07.675208 24894 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:46:07.710471 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45531
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:43659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:07.820469 24906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:07.820770 24906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:07.820814 24906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:07.824843 24906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:07.824909 24906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:07.824968 24906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:46:07.830350 24906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:43659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:07.831410 24906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:07.832373 24906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:07.841665 24911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:07.841671 24912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:07.841743 24906 server_base.cc:1034] running on GCE node
W20260929 18:46:07.841674 24914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:07.842264 24906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:07.842900 24906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:07.844058 24906 hybrid_clock.cc:648] HybridClock initialized: now 1790707567844039 us; error 56 us; skew 500 ppm
Sep 29 18:46:07 dist-test-slave-smpn krb5kdc[24799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707567, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:07.846602 24906 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:46:07.847481 24906 webserver.cc:460] Webserver started at http://127.7.157.65:34889/ using document root <none> and password file <none>
I20260929 18:46:07.848018 24906 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:07.848083 24906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:07.848268 24906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:07.850277 24906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "25836e14d384409c87545733c2baf49d"
format_stamp: "Formatted at 2026-09-29 18:46:07 on dist-test-slave-smpn"
server_key: "33fbb6d44f133afb7cf94a3677daf2df"
server_key_iv: "de3e064bebd0e64a396a13be1a354838"
server_key_version: "encryptionkey@0"
I20260929 18:46:07.850805 24906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "25836e14d384409c87545733c2baf49d"
format_stamp: "Formatted at 2026-09-29 18:46:07 on dist-test-slave-smpn"
server_key: "33fbb6d44f133afb7cf94a3677daf2df"
server_key_iv: "de3e064bebd0e64a396a13be1a354838"
server_key_version: "encryptionkey@0"
I20260929 18:46:07.854528 24906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:46:07.857105 24921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:07.858399 24906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:46:07.858510 24906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "25836e14d384409c87545733c2baf49d"
format_stamp: "Formatted at 2026-09-29 18:46:07 on dist-test-slave-smpn"
server_key: "33fbb6d44f133afb7cf94a3677daf2df"
server_key_iv: "de3e064bebd0e64a396a13be1a354838"
server_key_version: "encryptionkey@0"
I20260929 18:46:07.858632 24906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:07.870873 24906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:07.873744 24906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:07.873914 24906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:07.874498 24906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:07.875349 24906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:07.875402 24906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:07.875445 24906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:07.875473 24906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:07.883989 24906 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:40945
I20260929 18:46:07.884025 25034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:40945 every 8 connection(s)
I20260929 18:46:07.884886 24906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:46:07.886783 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 24906
I20260929 18:46:07.886902 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:46:07.887230 7797 external_mini_cluster.cc:1428] Setting key 19d19cfe653910d156d3601c5df0d8f5
Sep 29 18:46:07 dist-test-slave-smpn krb5kdc[24799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707567, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
I20260929 18:46:07.899190 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:07.886754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:34279 (local address 127.7.157.126:45531)
0929 18:46:07.886895 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:07.886900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:07.887937 (+ 1037us) server_negotiation.cc:407] Connection header received
0929 18:46:07.889387 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:07.889391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:07.889445 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:07.889530 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:07.890785 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:07.891459 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:07.892727 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:07.893080 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:07.893193 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:07.895777 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:07.895782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:07.895785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:07.895818 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:07.897191 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:07.897685 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:07.897687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:07.897688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:07.897757 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:07.898048 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:07.898050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:07.898051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:07.898264 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:07.898338 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:07.898873 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 18:46:07.899002 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:46:07.900242 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:07.887272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45531 (local address 127.7.157.65:34279)
0929 18:46:07.887772 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:07.887831 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:07.889160 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:07.889687 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:07.889694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:07.890107 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:07.890621 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:07.890635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:07.891602 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:07.891604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:07.892585 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:07.892595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:07.893235 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:07.893238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:07.893394 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:07.894052 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:07.894069 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:07.895576 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:07.897322 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:07.897327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:07.897341 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:07.897607 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:07.897825 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:07.897826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:07.897828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:07.897958 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:07.898352 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:07.898358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:07.898626 (+ 268us) client_negotiation.cc:769] Sending connection context
0929 18:46:07.898810 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 18:46:07.899155 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":125,"threads_started":1}
I20260929 18:46:07.901556 25035 heartbeater.cc:347] Connected to a master server at 127.7.157.126:45531
I20260929 18:46:07.901782 25035 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:07.902288 25035 heartbeater.cc:511] Master 127.7.157.126:45531 requested a full tablet report, sending...
I20260929 18:46:07.904079 24844 ts_manager.cc:194] Registered new tserver with Master: 25836e14d384409c87545733c2baf49d (127.7.157.65:40945)
I20260929 18:46:07.905664 24844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:34279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:46:07.914368 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45531
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:43659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:08.024420 25040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:08.024674 25040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:08.024716 25040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:08.028671 25040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:08.028731 25040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:08.028788 25040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:46:08.033968 25040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:43659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:08.034935 25040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:08.035952 25040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:08.045305 25046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:08.045447 25040 server_base.cc:1034] running on GCE node
W20260929 18:46:08.045315 25048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:08.045318 25045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:08.045977 25040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:08.046574 25040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:08.047706 25040 hybrid_clock.cc:648] HybridClock initialized: now 1790707568047689 us; error 30 us; skew 500 ppm
Sep 29 18:46:08 dist-test-slave-smpn krb5kdc[24799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707568, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:08.050325 25040 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:46:08.051236 25040 webserver.cc:460] Webserver started at http://127.7.157.66:40509/ using document root <none> and password file <none>
I20260929 18:46:08.051787 25040 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:08.051854 25040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:08.052053 25040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:08.054077 25040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "a42663a5e0c748cb8e86156ff272aec2"
format_stamp: "Formatted at 2026-09-29 18:46:08 on dist-test-slave-smpn"
server_key: "5fe9d2ceb9db025681ed7fcd595aac22"
server_key_iv: "d6a9bf69a2bf63a27c253019134e0404"
server_key_version: "encryptionkey@0"
I20260929 18:46:08.054622 25040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "a42663a5e0c748cb8e86156ff272aec2"
format_stamp: "Formatted at 2026-09-29 18:46:08 on dist-test-slave-smpn"
server_key: "5fe9d2ceb9db025681ed7fcd595aac22"
server_key_iv: "d6a9bf69a2bf63a27c253019134e0404"
server_key_version: "encryptionkey@0"
I20260929 18:46:08.058270 25040 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:46:08.060737 25055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:08.062019 25040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:46:08.062206 25040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "a42663a5e0c748cb8e86156ff272aec2"
format_stamp: "Formatted at 2026-09-29 18:46:08 on dist-test-slave-smpn"
server_key: "5fe9d2ceb9db025681ed7fcd595aac22"
server_key_iv: "d6a9bf69a2bf63a27c253019134e0404"
server_key_version: "encryptionkey@0"
I20260929 18:46:08.062331 25040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:08.079487 25040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:08.082576 25040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:08.082757 25040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:08.083374 25040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:08.084196 25040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:08.084242 25040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:08.084295 25040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:08.084323 25040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:08.092891 25040 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:38461
I20260929 18:46:08.092924 25168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:38461 every 8 connection(s)
I20260929 18:46:08.093786 25040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:46:08.100796 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 25040
I20260929 18:46:08.100921 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:46:08.101265 7797 external_mini_cluster.cc:1428] Setting key 75c3f8e493f1287cabc755e773708608
Sep 29 18:46:08 dist-test-slave-smpn krb5kdc[24799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707568, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:08.107860 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.095681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:53727 (local address 127.7.157.126:45531)
0929 18:46:08.095862 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:08.095866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:08.096728 (+ 862us) server_negotiation.cc:407] Connection header received
0929 18:46:08.098001 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:08.098004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:08.098058 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:08.098130 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:08.099326 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.099843 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.101033 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.101358 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.101482 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.104290 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:08.104293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:08.104296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:08.104321 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:08.105663 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:08.106223 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:08.106226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:08.106227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:08.106297 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:08.106639 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:08.106641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:08.106643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:08.106846 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:08.106929 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:08.107623 (+ 694us) server_negotiation.cc:299] Negotiation successful
0929 18:46:08.107736 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 18:46:08.109114 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.096061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45531 (local address 127.7.157.66:53727)
0929 18:46:08.096554 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:08.096602 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:08.097790 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:08.098304 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:08.098311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:08.098673 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:08.099194 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:08.099209 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.099965 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.099968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:08.100904 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:08.100915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.101546 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.101549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:08.101705 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.102297 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:08.102316 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:08.104117 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:08.105768 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:08.105774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:08.105791 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:08.106134 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:08.106369 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:08.106372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:08.106374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:08.106557 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:08.106963 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:08.106970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:08.107354 (+ 384us) client_negotiation.cc:769] Sending connection context
0929 18:46:08.107596 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 18:46:08.107972 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":128,"threads_started":1}
I20260929 18:46:08.110322 25169 heartbeater.cc:347] Connected to a master server at 127.7.157.126:45531
I20260929 18:46:08.110585 25169 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
I20260929 18:46:08.111176 25169 heartbeater.cc:511] Master 127.7.157.126:45531 requested a full tablet report, sending...
I20260929 18:46:08.112229 24844 ts_manager.cc:194] Registered new tserver with Master: a42663a5e0c748cb8e86156ff272aec2 (127.7.157.66:38461)
I20260929 18:46:08.112792 24844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:53727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:46:08.129804 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45531
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:43659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:08.238248 25174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:08.238525 25174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:08.238581 25174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:08.242699 25174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:08.242762 25174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:08.242851 25174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:46:08.248162 25174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:43659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:45531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:08.249258 25174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:08.250293 25174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:46:08.259896 25174 server_base.cc:1034] running on GCE node
W20260929 18:46:08.259770 25179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:08.259776 25182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:08.259766 25180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:08.260504 25174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:08.261230 25174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:08.262385 25174 hybrid_clock.cc:648] HybridClock initialized: now 1790707568262357 us; error 42 us; skew 500 ppm
Sep 29 18:46:08 dist-test-slave-smpn krb5kdc[24799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707568, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:08.265021 25174 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:46:08.265945 25174 webserver.cc:460] Webserver started at http://127.7.157.67:46277/ using document root <none> and password file <none>
I20260929 18:46:08.266561 25174 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:08.266628 25174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:08.266829 25174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:08.268931 25174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "46978e1b9ab34e6f8ee48e09112cb910"
format_stamp: "Formatted at 2026-09-29 18:46:08 on dist-test-slave-smpn"
server_key: "930f5e698447a9ed69e1f05601375850"
server_key_iv: "6271095b41cce416d6e80f503af1a24b"
server_key_version: "encryptionkey@0"
I20260929 18:46:08.269481 25174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "46978e1b9ab34e6f8ee48e09112cb910"
format_stamp: "Formatted at 2026-09-29 18:46:08 on dist-test-slave-smpn"
server_key: "930f5e698447a9ed69e1f05601375850"
server_key_iv: "6271095b41cce416d6e80f503af1a24b"
server_key_version: "encryptionkey@0"
I20260929 18:46:08.273257 25174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:46:08.275777 25189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:08.277009 25174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:08.277122 25174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "46978e1b9ab34e6f8ee48e09112cb910"
format_stamp: "Formatted at 2026-09-29 18:46:08 on dist-test-slave-smpn"
server_key: "930f5e698447a9ed69e1f05601375850"
server_key_iv: "6271095b41cce416d6e80f503af1a24b"
server_key_version: "encryptionkey@0"
I20260929 18:46:08.277223 25174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:08.289708 25174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:08.292472 25174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:08.292655 25174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:08.293347 25174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:08.294346 25174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:08.294402 25174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:08.294456 25174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:08.294486 25174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:08.304678 25174 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:44647
I20260929 18:46:08.304983 25302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:44647 every 8 connection(s)
I20260929 18:46:08.305634 25174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:46:08 dist-test-slave-smpn krb5kdc[24799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707568, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:08.315409 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 25174
I20260929 18:46:08.315517 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:46:08.315773 7797 external_mini_cluster.cc:1428] Setting key b9257443ae6d83c743cbda7c2b1d727a
I20260929 18:46:08.318059 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.307539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:59093 (local address 127.7.157.126:45531)
0929 18:46:08.307704 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:08.307709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:08.308571 (+ 862us) server_negotiation.cc:407] Connection header received
0929 18:46:08.309417 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:08.309419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:08.309466 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:08.309536 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:08.310676 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.311274 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.312344 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.312543 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.312609 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.314771 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:08.314774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:08.314776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:08.314802 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:08.316189 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:08.316659 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:08.316663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:08.316664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:08.316738 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:08.317046 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:08.317049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:08.317051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:08.317220 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:08.317305 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:08.317792 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 18:46:08.317903 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:46:08.318838 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.307961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45531 (local address 127.7.157.67:59093)
0929 18:46:08.308412 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:08.308457 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:08.309229 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:08.309674 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:08.309680 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:08.310067 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:08.310558 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:08.310572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.311384 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.311386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:08.312255 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:08.312264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.312650 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.312653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:08.312781 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.313297 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:08.313313 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:08.314610 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:08.316297 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:08.316303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:08.316318 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:08.316563 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:08.316819 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:08.316823 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:08.316825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:08.316957 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:08.317322 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:08.317329 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:08.317567 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 18:46:08.317732 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 18:46:08.318052 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":118,"threads_started":1}
I20260929 18:46:08.319842 25303 heartbeater.cc:347] Connected to a master server at 127.7.157.126:45531
I20260929 18:46:08.320081 25303 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:08.320531 25303 heartbeater.cc:511] Master 127.7.157.126:45531 requested a full tablet report, sending...
I20260929 18:46:08.321485 24844 ts_manager.cc:194] Registered new tserver with Master: 46978e1b9ab34e6f8ee48e09112cb910 (127.7.157.67:44647)
I20260929 18:46:08.322009 24844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:59093
I20260929 18:46:08.329329 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:46:08.338127 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.331217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.7.157.126:45531)
0929 18:46:08.331341 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:08.331344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:08.331521 (+ 177us) server_negotiation.cc:407] Connection header received
0929 18:46:08.331616 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:08.331618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:08.331656 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:08.331722 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:08.332369 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.332914 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.333716 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.333945 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.334028 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.335335 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:08.335339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:08.335342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:08.335367 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:08.336700 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:08.337072 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:08.337075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:08.337076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:08.337159 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:08.337364 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:08.337366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:08.337367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:08.337594 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:08.337697 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:08.337859 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 18:46:08.337991 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:46:08.340860 24844 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:46:08.343833 24844 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:46:08.357766 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.352723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:44647 (local address 127.0.0.1:55820)
0929 18:46:08.353145 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:08.353157 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:08.353243 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:08.353948 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:08.353951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:08.353968 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:08.354046 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:08.354052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.355369 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.355373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:08.356478 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:08.356486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.357245 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.357249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:08.357356 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.357393 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 18:46:08.357497 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 18:46:08.357611 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":90,"threads_started":1}
I20260929 18:46:08.357770 25319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.353118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.7.157.67:44647)
0929 18:46:08.353560 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:08.353571 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:08.353594 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:46:08.353666 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:08.353671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:08.353823 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:08.353966 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:08.354153 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.355245 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.356597 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.357130 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.357225 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.357371 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:08.357539 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 18:46:08.357630 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":132,"threads_started":1}
I20260929 18:46:08.358016 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.351970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:38461 (local address 127.0.0.1:60082)
0929 18:46:08.352611 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:08.352624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:08.352709 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:08.354122 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:08.354124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:08.354140 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:08.354198 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:08.354202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.355187 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.355191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:08.356274 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:08.356283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.357509 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.357512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:08.357692 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.357745 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 18:46:08.357812 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:46:08.357880 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":52,"thread_start_us":93,"threads_started":1}
I20260929 18:46:08.358104 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.351979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42244 (local address 127.7.157.65:40945)
0929 18:46:08.352732 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:08.352744 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:08.352774 (+ 30us) server_negotiation.cc:407] Connection header received
0929 18:46:08.352850 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:08.352855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:08.353082 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:08.353238 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:08.353705 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.354855 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.356434 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.357210 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.357313 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.357445 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:08.357875 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 18:46:08.357966 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":134,"threads_started":1}
I20260929 18:46:08.358413 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.351860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:40945 (local address 127.0.0.1:42244)
0929 18:46:08.352198 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:08.352244 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:08.352364 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:08.353407 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:08.353413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:08.353449 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:08.353578 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:08.353585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.354999 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.355002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:08.356331 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:08.356342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.357509 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.357512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:08.357693 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.357745 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 18:46:08.358220 (+ 475us) client_negotiation.cc:240] Negotiation successful
0929 18:46:08.358295 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":68,"thread_start_us":84,"threads_started":1}
I20260929 18:46:08.358444 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.352659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.7.157.66:38461)
0929 18:46:08.353519 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:08.353540 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:08.353559 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:46:08.353616 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:08.353620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:08.353807 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:08.353956 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:08.354302 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.355052 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.356392 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.357210 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.357321 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.357445 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:08.358235 (+ 790us) server_negotiation.cc:299] Negotiation successful
0929 18:46:08.358319 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":148,"threads_started":1}
I20260929 18:46:08.361001 25103 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1a00860fea4e09a30d99ab0fd8ee8f (DEFAULT_TABLE table=test-table [id=99b702e8cb8245e8ae26c65f1b03e41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:08.361058 25235 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1a00860fea4e09a30d99ab0fd8ee8f (DEFAULT_TABLE table=test-table [id=99b702e8cb8245e8ae26c65f1b03e41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:08.361336 24969 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1a00860fea4e09a30d99ab0fd8ee8f (DEFAULT_TABLE table=test-table [id=99b702e8cb8245e8ae26c65f1b03e41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:08.361990 25103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1a00860fea4e09a30d99ab0fd8ee8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:08.362084 25235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1a00860fea4e09a30d99ab0fd8ee8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:08.362170 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1a00860fea4e09a30d99ab0fd8ee8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:08.368398 25320 tablet_bootstrap.cc:492] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2: Bootstrap starting.
I20260929 18:46:08.369863 25321 tablet_bootstrap.cc:492] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d: Bootstrap starting.
I20260929 18:46:08.371402 25322 tablet_bootstrap.cc:492] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910: Bootstrap starting.
I20260929 18:46:08.371788 25320 tablet_bootstrap.cc:654] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:08.372044 25321 tablet_bootstrap.cc:654] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:08.372731 25320 log.cc:824] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:08.373210 25321 log.cc:824] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:08.373669 25322 tablet_bootstrap.cc:654] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:08.374313 25322 log.cc:824] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:08.374758 25320 tablet_bootstrap.cc:492] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2: No bootstrap required, opened a new log
I20260929 18:46:08.374984 25320 ts_tablet_manager.cc:1397] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 18:46:08.376066 25322 tablet_bootstrap.cc:492] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910: No bootstrap required, opened a new log
I20260929 18:46:08.376236 25322 ts_tablet_manager.cc:1397] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:46:08.378060 25320 raft_consensus.cc:348] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25836e14d384409c87545733c2baf49d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40945 } } peers { permanent_uuid: "a42663a5e0c748cb8e86156ff272aec2" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38461 } } peers { permanent_uuid: "46978e1b9ab34e6f8ee48e09112cb910" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 44647 } }
I20260929 18:46:08.378286 25320 raft_consensus.cc:374] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:08.378319 25320 raft_consensus.cc:729] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a42663a5e0c748cb8e86156ff272aec2, State: Initialized, Role: FOLLOWER
I20260929 18:46:08.378715 25320 consensus_queue.cc:260] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25836e14d384409c87545733c2baf49d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40945 } } peers { permanent_uuid: "a42663a5e0c748cb8e86156ff272aec2" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38461 } } peers { permanent_uuid: "46978e1b9ab34e6f8ee48e09112cb910" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 44647 } }
I20260929 18:46:08.379364 25169 heartbeater.cc:503] Master 127.7.157.126:45531 was elected leader, sending a full tablet report...
I20260929 18:46:08.379267 25322 raft_consensus.cc:348] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25836e14d384409c87545733c2baf49d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40945 } } peers { permanent_uuid: "a42663a5e0c748cb8e86156ff272aec2" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38461 } } peers { permanent_uuid: "46978e1b9ab34e6f8ee48e09112cb910" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 44647 } }
I20260929 18:46:08.379536 25322 raft_consensus.cc:374] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:08.379585 25322 raft_consensus.cc:729] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46978e1b9ab34e6f8ee48e09112cb910, State: Initialized, Role: FOLLOWER
I20260929 18:46:08.379750 25320 ts_tablet_manager.cc:1428] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:46:08.379979 25322 consensus_queue.cc:260] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25836e14d384409c87545733c2baf49d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40945 } } peers { permanent_uuid: "a42663a5e0c748cb8e86156ff272aec2" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38461 } } peers { permanent_uuid: "46978e1b9ab34e6f8ee48e09112cb910" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 44647 } }
I20260929 18:46:08.380766 25321 tablet_bootstrap.cc:492] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d: No bootstrap required, opened a new log
I20260929 18:46:08.380915 25322 ts_tablet_manager.cc:1428] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260929 18:46:08.380962 25321 ts_tablet_manager.cc:1397] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260929 18:46:08.381008 25303 heartbeater.cc:503] Master 127.7.157.126:45531 was elected leader, sending a full tablet report...
I20260929 18:46:08.384757 25321 raft_consensus.cc:348] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25836e14d384409c87545733c2baf49d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40945 } } peers { permanent_uuid: "a42663a5e0c748cb8e86156ff272aec2" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38461 } } peers { permanent_uuid: "46978e1b9ab34e6f8ee48e09112cb910" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 44647 } }
I20260929 18:46:08.384984 25321 raft_consensus.cc:374] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:08.385030 25321 raft_consensus.cc:729] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25836e14d384409c87545733c2baf49d, State: Initialized, Role: FOLLOWER
I20260929 18:46:08.385424 25321 consensus_queue.cc:260] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25836e14d384409c87545733c2baf49d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40945 } } peers { permanent_uuid: "a42663a5e0c748cb8e86156ff272aec2" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38461 } } peers { permanent_uuid: "46978e1b9ab34e6f8ee48e09112cb910" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 44647 } }
I20260929 18:46:08.386068 25035 heartbeater.cc:503] Master 127.7.157.126:45531 was elected leader, sending a full tablet report...
I20260929 18:46:08.386272 25321 ts_tablet_manager.cc:1428] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260929 18:46:08.556614 25304 tablet.cc:2369] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:46:08.594884 25170 tablet.cc:2369] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:46:08.636416 25036 tablet.cc:2369] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:46:08.683655 25326 raft_consensus.cc:482] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:46:08.683881 25326 raft_consensus.cc:504] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25836e14d384409c87545733c2baf49d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40945 } } peers { permanent_uuid: "a42663a5e0c748cb8e86156ff272aec2" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38461 } } peers { permanent_uuid: "46978e1b9ab34e6f8ee48e09112cb910" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 44647 } }
I20260929 18:46:08.685022 25326 leader_election.cc:288] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25836e14d384409c87545733c2baf49d (127.7.157.65:40945), 46978e1b9ab34e6f8ee48e09112cb910 (127.7.157.67:44647)
I20260929 18:46:08.689131 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.685244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:40945 (local address 127.7.157.66:45789)
0929 18:46:08.685487 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:08.685500 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:08.685615 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:08.685897 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:08.685900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:08.685925 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:08.686075 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:08.686082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.686938 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.686941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:08.688026 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:08.688036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.688679 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.688683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:08.688806 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.688830 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:46:08.688889 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 18:46:08.688950 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":71,"spinlock_wait_cycles":18816,"thread_start_us":70,"threads_started":1}
I20260929 18:46:08.689132 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.685344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:45789 (local address 127.7.157.65:40945)
0929 18:46:08.685536 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:08.685541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:08.685562 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:46:08.685729 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:08.685732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:08.685787 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:08.685864 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:08.686192 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.686830 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.688142 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.688572 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.688659 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.688702 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:08.688932 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 18:46:08.689008 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 18:46:08.689146 25319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.685756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:43423 (local address 127.7.157.67:44647)
0929 18:46:08.685886 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:08.685890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:08.685926 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:46:08.686087 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:08.686089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:08.686138 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:08.686208 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:08.686483 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.687034 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.688209 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.688637 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.688747 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.688794 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:08.688977 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:46:08.689046 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:46:08.689162 25330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.685594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:44647 (local address 127.7.157.66:43423)
0929 18:46:08.685860 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:08.685872 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:08.685983 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:08.686231 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:08.686233 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:08.686251 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:08.686377 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:08.686383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.687169 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.687172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:08.688092 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:08.688103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.688757 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.688761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:08.688858 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.688877 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:46:08.688950 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:46:08.689010 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":3,"spinlock_wait_cycles":2560,"thread_start_us":79,"threads_started":1}
I20260929 18:46:08.689795 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a00860fea4e09a30d99ab0fd8ee8f" candidate_uuid: "a42663a5e0c748cb8e86156ff272aec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25836e14d384409c87545733c2baf49d" is_pre_election: true
I20260929 18:46:08.690104 24989 raft_consensus.cc:2454] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a42663a5e0c748cb8e86156ff272aec2 in term 0.
I20260929 18:46:08.690557 25056 leader_election.cc:302] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25836e14d384409c87545733c2baf49d, a42663a5e0c748cb8e86156ff272aec2; no voters:
I20260929 18:46:08.690760 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a00860fea4e09a30d99ab0fd8ee8f" candidate_uuid: "a42663a5e0c748cb8e86156ff272aec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46978e1b9ab34e6f8ee48e09112cb910" is_pre_election: true
I20260929 18:46:08.690838 25326 raft_consensus.cc:2781] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:46:08.690907 25326 raft_consensus.cc:482] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:46:08.690963 25326 raft_consensus.cc:3037] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:08.691047 25247 raft_consensus.cc:2454] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a42663a5e0c748cb8e86156ff272aec2 in term 0.
I20260929 18:46:08.692116 25326 raft_consensus.cc:504] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25836e14d384409c87545733c2baf49d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40945 } } peers { permanent_uuid: "a42663a5e0c748cb8e86156ff272aec2" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38461 } } peers { permanent_uuid: "46978e1b9ab34e6f8ee48e09112cb910" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 44647 } }
I20260929 18:46:08.692435 25326 leader_election.cc:288] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [CANDIDATE]: Term 1 election: Requested vote from peers 25836e14d384409c87545733c2baf49d (127.7.157.65:40945), 46978e1b9ab34e6f8ee48e09112cb910 (127.7.157.67:44647)
I20260929 18:46:08.692785 25247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a00860fea4e09a30d99ab0fd8ee8f" candidate_uuid: "a42663a5e0c748cb8e86156ff272aec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46978e1b9ab34e6f8ee48e09112cb910"
I20260929 18:46:08.692787 24989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1a00860fea4e09a30d99ab0fd8ee8f" candidate_uuid: "a42663a5e0c748cb8e86156ff272aec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25836e14d384409c87545733c2baf49d"
I20260929 18:46:08.692888 25247 raft_consensus.cc:3037] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:08.692896 24989 raft_consensus.cc:3037] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:08.693889 25247 raft_consensus.cc:2454] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a42663a5e0c748cb8e86156ff272aec2 in term 1.
I20260929 18:46:08.693903 24989 raft_consensus.cc:2454] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a42663a5e0c748cb8e86156ff272aec2 in term 1.
I20260929 18:46:08.694164 25056 leader_election.cc:302] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25836e14d384409c87545733c2baf49d, a42663a5e0c748cb8e86156ff272aec2; no voters:
I20260929 18:46:08.694335 25326 raft_consensus.cc:2781] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:08.694533 25326 raft_consensus.cc:686] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [term 1 LEADER]: Becoming Leader. State: Replica: a42663a5e0c748cb8e86156ff272aec2, State: Running, Role: LEADER
I20260929 18:46:08.694873 25326 consensus_queue.cc:237] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25836e14d384409c87545733c2baf49d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40945 } } peers { permanent_uuid: "a42663a5e0c748cb8e86156ff272aec2" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38461 } } peers { permanent_uuid: "46978e1b9ab34e6f8ee48e09112cb910" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 44647 } }
I20260929 18:46:08.697844 24843 catalog_manager.cc:5495] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a42663a5e0c748cb8e86156ff272aec2 (127.7.157.66). New cstate: current_term: 1 leader_uuid: "a42663a5e0c748cb8e86156ff272aec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25836e14d384409c87545733c2baf49d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a42663a5e0c748cb8e86156ff272aec2" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46978e1b9ab34e6f8ee48e09112cb910" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20260929 18:46:08.728334 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.725159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60090 (local address 127.7.157.66:38461)
0929 18:46:08.725292 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:08.725296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:08.725308 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:46:08.725382 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:08.725385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:08.725441 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:08.725547 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:46:08.725750 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.726244 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.727032 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.727258 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.727331 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.727581 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:46:08.728009 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:46:08.728099 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:08.728144 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 18:46:08.728195 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:46:08.734432 24989 raft_consensus.cc:1261] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d [term 1 FOLLOWER]: Refusing update from remote peer a42663a5e0c748cb8e86156ff272aec2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:46:08.734432 25247 raft_consensus.cc:1261] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910 [term 1 FOLLOWER]: Refusing update from remote peer a42663a5e0c748cb8e86156ff272aec2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:46:08.735199 25326 consensus_queue.cc:1035] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25836e14d384409c87545733c2baf49d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:46:08.735399 25331 consensus_queue.cc:1035] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46978e1b9ab34e6f8ee48e09112cb910" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:46:08.752658 24843 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:46:08.752815 24843 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:46:08.755138 24843 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be7573182f79400790e17f4ed2d0ffec: Sending DeleteTablet for 3 replicas of tablet 3e1a00860fea4e09a30d99ab0fd8ee8f
I20260929 18:46:08.755967 24969 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e1a00860fea4e09a30d99ab0fd8ee8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:08 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:42244
I20260929 18:46:08.756057 25235 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e1a00860fea4e09a30d99ab0fd8ee8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:08 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:55820
I20260929 18:46:08.756457 25344 tablet_replica.cc:330] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d: stopping tablet replica
I20260929 18:46:08.756486 25103 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e1a00860fea4e09a30d99ab0fd8ee8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:08 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:60082
I20260929 18:46:08.756762 25344 raft_consensus.cc:2229] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:46:08.756984 25346 tablet_replica.cc:330] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2: stopping tablet replica
I20260929 18:46:08.757085 25344 raft_consensus.cc:2258] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 25836e14d384409c87545733c2baf49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:46:08.757087 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 24906
I20260929 18:46:08.757433 25345 tablet_replica.cc:330] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910: stopping tablet replica
I20260929 18:46:08.757716 25345 raft_consensus.cc:2229] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:46:08.757843 25346 raft_consensus.cc:2229] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:46:08.758000 25345 raft_consensus.cc:2258] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:46:08.758718 25346 raft_consensus.cc:2258] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:46:08.759410 25345 ts_tablet_manager.cc:1905] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:46:08.760162 25346 ts_tablet_manager.cc:1905] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:46:08.762418 25345 ts_tablet_manager.cc:1918] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:46:08.762519 25345 log.cc:1196] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/3e1a00860fea4e09a30d99ab0fd8ee8f
I20260929 18:46:08.762830 25345 ts_tablet_manager.cc:1939] T 3e1a00860fea4e09a30d99ab0fd8ee8f P 46978e1b9ab34e6f8ee48e09112cb910: Deleting consensus metadata
I20260929 18:46:08.762859 25346 ts_tablet_manager.cc:1918] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:46:08.763060 25346 log.cc:1196] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/3e1a00860fea4e09a30d99ab0fd8ee8f
I20260929 18:46:08.763402 25346 ts_tablet_manager.cc:1939] T 3e1a00860fea4e09a30d99ab0fd8ee8f P a42663a5e0c748cb8e86156ff272aec2: Deleting consensus metadata
I20260929 18:46:08.764196 24828 catalog_manager.cc:4841] TS 46978e1b9ab34e6f8ee48e09112cb910 (127.7.157.67:44647): tablet 3e1a00860fea4e09a30d99ab0fd8ee8f (table test-table [id=99b702e8cb8245e8ae26c65f1b03e41b]) successfully deleted
I20260929 18:46:08.764729 24831 catalog_manager.cc:4841] TS a42663a5e0c748cb8e86156ff272aec2 (127.7.157.66:38461): tablet 3e1a00860fea4e09a30d99ab0fd8ee8f (table test-table [id=99b702e8cb8245e8ae26c65f1b03e41b]) successfully deleted
W20260929 18:46:08.766986 24828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:40945 (error 108)
I20260929 18:46:08.767158 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 25040
I20260929 18:46:08.769192 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.768886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:40945 (local address 127.0.0.1:42250)
0929 18:46:08.769028 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:08.769103 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:40945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 18:46:08.769362 24828 catalog_manager.cc:4584] TS 25836e14d384409c87545733c2baf49d (127.7.157.65:40945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e1a00860fea4e09a30d99ab0fd8ee8f: Network error: Client connection negotiation failed: client connection to 127.7.157.65:40945: connect: Connection refused (error 111)
I20260929 18:46:08.772521 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 25174
I20260929 18:46:08.777187 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 24812
2026-09-29T18:46:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2414 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T18:46:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:08Z Disabled control of system clock
I20260929 18:46:08.804607 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:39591
--webserver_interface=127.7.157.126
--webserver_port=0
--builtin_ntp_servers=127.7.157.84:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:39591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:46:08.910385 25351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:08.910669 25351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:08.910706 25351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:08.914734 25351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:08.914806 25351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:08.914820 25351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:08.914836 25351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:08.914849 25351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:08.919656 25351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:39591
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:39591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:08.920714 25351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:08.922029 25351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:08.930362 25356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:08.930429 25351 server_base.cc:1034] running on GCE node
W20260929 18:46:08.930588 25359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:08.930374 25357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:08.931177 25351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:08.932296 25351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:08.933456 25351 hybrid_clock.cc:648] HybridClock initialized: now 1790707568933436 us; error 33 us; skew 500 ppm
I20260929 18:46:08.935315 25351 webserver.cc:460] Webserver started at http://127.7.157.126:41547/ using document root <none> and password file <none>
I20260929 18:46:08.935853 25351 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:08.935910 25351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:08.936105 25351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:08.938171 25351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "17005dc5e9a849f0a9dde571d1d94093"
format_stamp: "Formatted at 2026-09-29 18:46:08 on dist-test-slave-smpn"
server_key: "90cefa8beaf45a593831d21898d3e439"
server_key_iv: "6c36afff70e86a5231f2db82ed215e2a"
server_key_version: "encryptionkey@0"
I20260929 18:46:08.938691 25351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "17005dc5e9a849f0a9dde571d1d94093"
format_stamp: "Formatted at 2026-09-29 18:46:08 on dist-test-slave-smpn"
server_key: "90cefa8beaf45a593831d21898d3e439"
server_key_iv: "6c36afff70e86a5231f2db82ed215e2a"
server_key_version: "encryptionkey@0"
I20260929 18:46:08.942579 25351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:46:08.945008 25365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:08.946261 25351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:46:08.946363 25351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "17005dc5e9a849f0a9dde571d1d94093"
format_stamp: "Formatted at 2026-09-29 18:46:08 on dist-test-slave-smpn"
server_key: "90cefa8beaf45a593831d21898d3e439"
server_key_iv: "6c36afff70e86a5231f2db82ed215e2a"
server_key_version: "encryptionkey@0"
I20260929 18:46:08.946451 25351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:08.973371 25351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:08.974080 25351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:08.974236 25351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:08.980875 25351 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:39591
I20260929 18:46:08.980894 25417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:39591 every 8 connection(s)
I20260929 18:46:08.981846 25351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:08.984944 25418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:08.989880 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 25351
I20260929 18:46:08.989982 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:08.990228 7797 external_mini_cluster.cc:1428] Setting key bae4d0a1c0de7073121bf832b2f9ce13
I20260929 18:46:08.990404 25418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093: Bootstrap starting.
I20260929 18:46:08.992870 25418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:08.993578 25418 log.cc:824] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:08.995473 25418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093: No bootstrap required, opened a new log
I20260929 18:46:08.999032 25418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17005dc5e9a849f0a9dde571d1d94093" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39591 } }
I20260929 18:46:08.999300 25418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:08.999358 25418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17005dc5e9a849f0a9dde571d1d94093, State: Initialized, Role: FOLLOWER
I20260929 18:46:08.999465 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:08.991613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.7.157.126:39591)
0929 18:46:08.992258 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:08.992271 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:08.992307 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:46:08.992906 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:08.992922 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:08.993358 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:08.993667 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:08.994094 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.995046 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.996018 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:08.996548 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:08.996707 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:08.997608 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:08.997614 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:08.997627 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:08.998029 (+ 402us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:08.998089 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:08.998109 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:08.998194 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:08.998357 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 18:46:08.998616 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":117,"threads_started":1}
I20260929 18:46:08.999852 25418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17005dc5e9a849f0a9dde571d1d94093" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39591 } }
I20260929 18:46:09.000017 25418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:09.000082 25418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:09.000172 25418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:09.001142 25418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17005dc5e9a849f0a9dde571d1d94093" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39591 } }
I20260929 18:46:09.001478 25418 leader_election.cc:302] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17005dc5e9a849f0a9dde571d1d94093; no voters:
I20260929 18:46:09.001821 25418 leader_election.cc:288] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:09.001924 25423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:09.002146 25423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [term 1 LEADER]: Becoming Leader. State: Replica: 17005dc5e9a849f0a9dde571d1d94093, State: Running, Role: LEADER
I20260929 18:46:09.002513 25423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17005dc5e9a849f0a9dde571d1d94093" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39591 } }
I20260929 18:46:09.002981 25418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:09.004086 25424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17005dc5e9a849f0a9dde571d1d94093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17005dc5e9a849f0a9dde571d1d94093" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39591 } } }
I20260929 18:46:09.004215 25424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:09.004520 25425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17005dc5e9a849f0a9dde571d1d94093. Latest consensus state: current_term: 1 leader_uuid: "17005dc5e9a849f0a9dde571d1d94093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17005dc5e9a849f0a9dde571d1d94093" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39591 } } }
I20260929 18:46:09.004633 25425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:09.004722 25432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:09.008253 25432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:09.013770 25432 catalog_manager.cc:1347] Generated new cluster ID: d4335bf4c58c4e46bae044d4c11f4bf8
I20260929 18:46:09.013845 25432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:09.028920 25432 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:09.030117 25432 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:09.044399 25432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093: Generated new TSK 0
I20260929 18:46:09.045251 25432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:09.053644 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:39591
--builtin_ntp_servers=127.7.157.84:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:46:09.163556 25442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:09.163802 25442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:09.163837 25442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:09.167848 25442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:09.167904 25442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:09.167958 25442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:46:09.172583 25442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:39591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:09.173641 25442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:09.174996 25442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:09.184468 25448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:09.184481 25447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:09.184544 25442 server_base.cc:1034] running on GCE node
W20260929 18:46:09.184645 25450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:09.185052 25442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:09.185671 25442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:09.186836 25442 hybrid_clock.cc:648] HybridClock initialized: now 1790707569186818 us; error 28 us; skew 500 ppm
I20260929 18:46:09.188553 25442 webserver.cc:460] Webserver started at http://127.7.157.65:42283/ using document root <none> and password file <none>
I20260929 18:46:09.189124 25442 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:09.189184 25442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:09.189381 25442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:09.191380 25442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "b86944431c4e477691c205914b18b69d"
format_stamp: "Formatted at 2026-09-29 18:46:09 on dist-test-slave-smpn"
server_key: "7be6cc3820a3b9f64d8df37c8bc9eb9f"
server_key_iv: "9bcf72ac99c6e01e223bd24b769be820"
server_key_version: "encryptionkey@0"
I20260929 18:46:09.191893 25442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "b86944431c4e477691c205914b18b69d"
format_stamp: "Formatted at 2026-09-29 18:46:09 on dist-test-slave-smpn"
server_key: "7be6cc3820a3b9f64d8df37c8bc9eb9f"
server_key_iv: "9bcf72ac99c6e01e223bd24b769be820"
server_key_version: "encryptionkey@0"
I20260929 18:46:09.195379 25442 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:46:09.197647 25456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:09.198776 25442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:46:09.198887 25442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "b86944431c4e477691c205914b18b69d"
format_stamp: "Formatted at 2026-09-29 18:46:09 on dist-test-slave-smpn"
server_key: "7be6cc3820a3b9f64d8df37c8bc9eb9f"
server_key_iv: "9bcf72ac99c6e01e223bd24b769be820"
server_key_version: "encryptionkey@0"
I20260929 18:46:09.198979 25442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:09.220758 25442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:09.221511 25442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:09.221704 25442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:09.222321 25442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:09.223337 25442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:09.223389 25442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:09.223445 25442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:09.223474 25442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:09.231842 25442 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:39539
I20260929 18:46:09.231874 25569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:39539 every 8 connection(s)
I20260929 18:46:09.232725 25442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:46:09.239771 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 25442
I20260929 18:46:09.239882 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:46:09.240180 7797 external_mini_cluster.cc:1428] Setting key 51cce6120a8993dc67a7d956a1e3c1b5
I20260929 18:46:09.242102 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.234447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:53649 (local address 127.7.157.126:39591)
0929 18:46:09.234590 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:09.234594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:09.235673 (+ 1079us) server_negotiation.cc:407] Connection header received
0929 18:46:09.237061 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:09.237064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:09.237123 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:09.237197 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:09.238135 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.238679 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.239943 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.240292 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.240411 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.241590 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:09.241593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:09.241595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:09.241607 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:09.241627 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:09.241635 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:09.241703 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:09.241866 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 18:46:09.241931 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:46:09.242368 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:39591
--builtin_ntp_servers=127.7.157.84:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:46:09.243167 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.234854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39591 (local address 127.7.157.65:53649)
0929 18:46:09.235521 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:09.235563 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:09.236828 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:09.237386 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:09.237393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:09.237488 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:09.237978 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:09.237990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.238809 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.238812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:09.239783 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:09.239794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.240450 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.240453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:09.240662 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.241278 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:46:09.241279 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:09.241337 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:46:09.241426 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:09.241718 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:09.241728 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:09.241734 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:46:09.241844 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 18:46:09.242137 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":143,"threads_started":1}
I20260929 18:46:09.244393 25570 heartbeater.cc:347] Connected to a master server at 127.7.157.126:39591
I20260929 18:46:09.244661 25570 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:09.245184 25570 heartbeater.cc:511] Master 127.7.157.126:39591 requested a full tablet report, sending...
I20260929 18:46:09.246762 25382 ts_manager.cc:194] Registered new tserver with Master: b86944431c4e477691c205914b18b69d (127.7.157.65:39539)
I20260929 18:46:09.248397 25382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.157.65:53649
W20260929 18:46:09.351316 25573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:09.351579 25573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:09.351616 25573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:09.355600 25573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:09.355659 25573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:09.355712 25573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:46:09.360389 25573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:39591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:09.361392 25573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:09.362660 25573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:09.371874 25578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:09.371865 25573 server_base.cc:1034] running on GCE node
W20260929 18:46:09.371841 25581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:09.371861 25579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:09.372509 25573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:09.373119 25573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:09.374272 25573 hybrid_clock.cc:648] HybridClock initialized: now 1790707569374256 us; error 27 us; skew 500 ppm
I20260929 18:46:09.376094 25573 webserver.cc:460] Webserver started at http://127.7.157.66:41835/ using document root <none> and password file <none>
I20260929 18:46:09.376644 25573 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:09.376705 25573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:09.376889 25573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:09.378875 25573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "759e8c7e73bc45348f1b0d718ac1b3d7"
format_stamp: "Formatted at 2026-09-29 18:46:09 on dist-test-slave-smpn"
server_key: "751256a6b42bc004ae4335dbb13fbcf6"
server_key_iv: "e5782a923c65ff5c5112b74c2597543b"
server_key_version: "encryptionkey@0"
I20260929 18:46:09.379449 25573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "759e8c7e73bc45348f1b0d718ac1b3d7"
format_stamp: "Formatted at 2026-09-29 18:46:09 on dist-test-slave-smpn"
server_key: "751256a6b42bc004ae4335dbb13fbcf6"
server_key_iv: "e5782a923c65ff5c5112b74c2597543b"
server_key_version: "encryptionkey@0"
I20260929 18:46:09.383147 25573 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:46:09.385582 25587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:09.386750 25573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:46:09.386857 25573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "759e8c7e73bc45348f1b0d718ac1b3d7"
format_stamp: "Formatted at 2026-09-29 18:46:09 on dist-test-slave-smpn"
server_key: "751256a6b42bc004ae4335dbb13fbcf6"
server_key_iv: "e5782a923c65ff5c5112b74c2597543b"
server_key_version: "encryptionkey@0"
I20260929 18:46:09.386952 25573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:09.397178 25573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:09.397810 25573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:09.397951 25573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:09.398473 25573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:09.399438 25573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:09.399489 25573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:09.399529 25573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:09.399549 25573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:09.408035 25573 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:45495
I20260929 18:46:09.408056 25700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:45495 every 8 connection(s)
I20260929 18:46:09.408910 25573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:46:09.409476 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 25573
I20260929 18:46:09.409614 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:46:09.409848 7797 external_mini_cluster.cc:1428] Setting key 5f387c8c9e01ea2e84691ff19b1596dc
I20260929 18:46:09.411916 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:39591
--builtin_ntp_servers=127.7.157.84:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:46:09.418251 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.410651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:45861 (local address 127.7.157.126:39591)
0929 18:46:09.410776 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:09.410779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:09.411962 (+ 1183us) server_negotiation.cc:407] Connection header received
0929 18:46:09.413438 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:09.413448 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:09.413495 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:09.413569 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:09.414701 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.415269 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.416405 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.416634 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.416722 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.417788 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:09.417791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:09.417793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:09.417806 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:09.417825 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:09.417833 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:09.417891 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:09.418037 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 18:46:09.418101 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:46:09.419198 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.411172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39591 (local address 127.7.157.66:45861)
0929 18:46:09.411821 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:09.411861 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:09.413208 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:09.413780 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:09.413789 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:09.413903 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:09.414572 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:09.414591 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.415386 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.415389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:09.416280 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:09.416290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.416784 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.416788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:09.416947 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.417522 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:46:09.417524 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:09.417566 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:46:09.417634 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:09.417906 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:09.417914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:09.417920 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:46:09.418036 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 18:46:09.418329 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":128,"threads_started":1}
I20260929 18:46:09.420322 25701 heartbeater.cc:347] Connected to a master server at 127.7.157.126:39591
I20260929 18:46:09.420572 25701 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:09.421097 25701 heartbeater.cc:511] Master 127.7.157.126:39591 requested a full tablet report, sending...
I20260929 18:46:09.422135 25382 ts_manager.cc:194] Registered new tserver with Master: 759e8c7e73bc45348f1b0d718ac1b3d7 (127.7.157.66:45495)
I20260929 18:46:09.422863 25382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.157.66:45861
W20260929 18:46:09.520337 25704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:09.520707 25704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:09.520773 25704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:09.524782 25704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:09.524839 25704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:09.524904 25704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:46:09.529670 25704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:39591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:09.530670 25704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:09.532037 25704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:46:09.541198 25704 server_base.cc:1034] running on GCE node
W20260929 18:46:09.541123 25709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:09.541141 25710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:09.541145 25712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:09.541764 25704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:09.542344 25704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:09.543493 25704 hybrid_clock.cc:648] HybridClock initialized: now 1790707569543474 us; error 31 us; skew 500 ppm
I20260929 18:46:09.545171 25704 webserver.cc:460] Webserver started at http://127.7.157.67:38879/ using document root <none> and password file <none>
I20260929 18:46:09.545711 25704 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:09.545770 25704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:09.545974 25704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:09.547991 25704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "a226e3c163524edd84f3b1be6f36c639"
format_stamp: "Formatted at 2026-09-29 18:46:09 on dist-test-slave-smpn"
server_key: "d65407733c2707089b2783b38416b7ee"
server_key_iv: "45f183f27389c134b6e0f31011d1a598"
server_key_version: "encryptionkey@0"
I20260929 18:46:09.548466 25704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "a226e3c163524edd84f3b1be6f36c639"
format_stamp: "Formatted at 2026-09-29 18:46:09 on dist-test-slave-smpn"
server_key: "d65407733c2707089b2783b38416b7ee"
server_key_iv: "45f183f27389c134b6e0f31011d1a598"
server_key_version: "encryptionkey@0"
I20260929 18:46:09.551970 25704 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:46:09.554203 25718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:09.555403 25704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:09.555509 25704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "a226e3c163524edd84f3b1be6f36c639"
format_stamp: "Formatted at 2026-09-29 18:46:09 on dist-test-slave-smpn"
server_key: "d65407733c2707089b2783b38416b7ee"
server_key_iv: "45f183f27389c134b6e0f31011d1a598"
server_key_version: "encryptionkey@0"
I20260929 18:46:09.555609 25704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:09.603276 25704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:09.604089 25704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:09.604327 25704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:09.604990 25704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:09.605971 25704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:09.606019 25704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:09.606065 25704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:09.606086 25704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:09.614248 25704 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:35439
I20260929 18:46:09.614279 25831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:35439 every 8 connection(s)
I20260929 18:46:09.615200 25704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:46:09.618587 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 25704
I20260929 18:46:09.618669 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:46:09.618903 7797 external_mini_cluster.cc:1428] Setting key fc7e2d59160d2d22b10da999ae3c9dc4
I20260929 18:46:09.623991 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.616937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:52189 (local address 127.7.157.126:39591)
0929 18:46:09.617077 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:09.617081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:09.618069 (+ 988us) server_negotiation.cc:407] Connection header received
0929 18:46:09.619493 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:09.619497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:09.619550 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:09.619639 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:09.620770 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.621340 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.622407 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.622617 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.622683 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.623655 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:09.623658 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:09.623660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:09.623670 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:09.623686 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:09.623693 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:09.623738 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:09.623845 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 18:46:09.623888 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:46:09.624819 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.617300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:39591 (local address 127.7.157.67:52189)
0929 18:46:09.617912 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:09.617952 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:09.619276 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:09.619846 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:09.619854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:09.619965 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:09.620629 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:09.620649 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.621443 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.621446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:09.622314 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:09.622323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.622726 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.622728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:09.622861 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.623422 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:46:09.623424 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:09.623462 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:46:09.623532 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:09.623749 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:09.623754 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:09.623757 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:46:09.623840 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:46:09.624048 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":112,"threads_started":1}
I20260929 18:46:09.625826 25832 heartbeater.cc:347] Connected to a master server at 127.7.157.126:39591
I20260929 18:46:09.626035 25832 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:09.626495 25832 heartbeater.cc:511] Master 127.7.157.126:39591 requested a full tablet report, sending...
I20260929 18:46:09.627408 25382 ts_manager.cc:194] Registered new tserver with Master: a226e3c163524edd84f3b1be6f36c639 (127.7.157.67:35439)
I20260929 18:46:09.627912 25382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.157.67:52189
I20260929 18:46:09.632449 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:46:09.637580 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.634278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53416 (local address 127.7.157.126:39591)
0929 18:46:09.634416 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:09.634419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:09.634494 (+ 75us) server_negotiation.cc:407] Connection header received
0929 18:46:09.634595 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:09.634597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:09.634632 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:09.634689 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:09.634886 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.635403 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.636208 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.636401 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.636463 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.637096 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:09.637099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:09.637101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:09.637111 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:09.637128 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:09.637135 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:09.637236 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:09.637327 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 18:46:09.637372 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 18:46:09.640087 25382 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:46:09.643011 25382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:46:09.657265 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.650911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:39539 (local address 127.0.0.1:53250)
0929 18:46:09.651603 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:09.651656 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:09.651793 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:09.652565 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:09.652570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:09.652593 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:09.652742 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:09.652750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.654149 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.654152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:09.655710 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:09.655721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.656668 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.656671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:09.656752 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.656812 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 18:46:09.656931 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 18:46:09.657089 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":94,"spinlock_wait_cycles":1920,"thread_start_us":103,"threads_started":1}
I20260929 18:46:09.657272 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.651114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53250 (local address 127.7.157.65:39539)
0929 18:46:09.651413 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:09.651425 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:09.651941 (+ 516us) server_negotiation.cc:407] Connection header received
0929 18:46:09.652039 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:09.652044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:09.652254 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:09.652401 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:09.652872 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.654030 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.655831 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.656554 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.656648 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.656802 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:09.656979 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 18:46:09.657091 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":90,"threads_started":1}
I20260929 18:46:09.657321 25848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.652110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.7.157.67:35439)
0929 18:46:09.652556 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:09.652573 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:09.652880 (+ 307us) server_negotiation.cc:407] Connection header received
0929 18:46:09.652955 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:09.652960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:09.653136 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:09.653268 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:09.653466 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.654222 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.655650 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.656421 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.656537 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.656736 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:09.657067 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 18:46:09.657174 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":162,"threads_started":1}
I20260929 18:46:09.657639 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.651713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:35439 (local address 127.0.0.1:38064)
0929 18:46:09.652780 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:09.652789 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:09.652858 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:09.653248 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:09.653251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:09.653269 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:09.653350 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:09.653357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.654339 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.654343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:09.655523 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:09.655533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.656532 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.656534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:09.656613 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.656873 (+ 260us) client_negotiation.cc:769] Sending connection context
0929 18:46:09.656931 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:46:09.657516 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":497,"thread_start_us":85,"threads_started":1}
I20260929 18:46:09.658385 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.650980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:45495 (local address 127.0.0.1:57788)
0929 18:46:09.652512 (+ 1532us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:09.652526 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:09.652634 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:09.653055 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:09.653058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:09.653075 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:09.653167 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:09.653174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.654419 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.654422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:09.655711 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:09.655721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.658034 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.658038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:09.658123 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.658144 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:46:09.658199 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 18:46:09.658260 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1469,"mutex_wait_us":54,"spinlock_wait_cycles":896,"thread_start_us":101,"threads_started":1}
I20260929 18:46:09.658473 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.651780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57788 (local address 127.7.157.66:45495)
0929 18:46:09.652105 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:09.652115 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:09.652622 (+ 507us) server_negotiation.cc:407] Connection header received
0929 18:46:09.652733 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:09.652738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:09.652926 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:09.653065 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:09.653294 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.654314 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.656857 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.657912 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.658013 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.658185 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:09.658244 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 18:46:09.658324 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":112,"threads_started":1}
I20260929 18:46:09.660571 25766 tablet_service.cc:1467] Processing CreateTablet for tablet 2a9d69a70c1642a08c197174c2124067 (DEFAULT_TABLE table=test-table [id=45b759a338334b399f6d6c9d2ea11573]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:09.661098 25635 tablet_service.cc:1467] Processing CreateTablet for tablet 2a9d69a70c1642a08c197174c2124067 (DEFAULT_TABLE table=test-table [id=45b759a338334b399f6d6c9d2ea11573]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:09.661553 25766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a9d69a70c1642a08c197174c2124067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:09.661964 25635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a9d69a70c1642a08c197174c2124067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:09.667699 25849 tablet_bootstrap.cc:492] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7: Bootstrap starting.
I20260929 18:46:09.669130 25850 tablet_bootstrap.cc:492] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639: Bootstrap starting.
I20260929 18:46:09.669255 25504 tablet_service.cc:1467] Processing CreateTablet for tablet 2a9d69a70c1642a08c197174c2124067 (DEFAULT_TABLE table=test-table [id=45b759a338334b399f6d6c9d2ea11573]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:09.669905 25849 tablet_bootstrap.cc:654] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:09.670346 25504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a9d69a70c1642a08c197174c2124067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:09.670635 25849 log.cc:824] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:09.671277 25850 tablet_bootstrap.cc:654] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:09.672122 25850 log.cc:824] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:09.672935 25849 tablet_bootstrap.cc:492] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7: No bootstrap required, opened a new log
I20260929 18:46:09.673136 25849 ts_tablet_manager.cc:1397] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 18:46:09.674379 25850 tablet_bootstrap.cc:492] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639: No bootstrap required, opened a new log
I20260929 18:46:09.674584 25850 ts_tablet_manager.cc:1397] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:46:09.676353 25849 raft_consensus.cc:348] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b86944431c4e477691c205914b18b69d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39539 } } peers { permanent_uuid: "759e8c7e73bc45348f1b0d718ac1b3d7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45495 } } peers { permanent_uuid: "a226e3c163524edd84f3b1be6f36c639" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35439 } }
I20260929 18:46:09.676589 25849 raft_consensus.cc:374] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:09.676667 25849 raft_consensus.cc:729] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759e8c7e73bc45348f1b0d718ac1b3d7, State: Initialized, Role: FOLLOWER
I20260929 18:46:09.677094 25849 consensus_queue.cc:260] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b86944431c4e477691c205914b18b69d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39539 } } peers { permanent_uuid: "759e8c7e73bc45348f1b0d718ac1b3d7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45495 } } peers { permanent_uuid: "a226e3c163524edd84f3b1be6f36c639" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35439 } }
I20260929 18:46:09.677534 25853 tablet_bootstrap.cc:492] T 2a9d69a70c1642a08c197174c2124067 P b86944431c4e477691c205914b18b69d: Bootstrap starting.
I20260929 18:46:09.678216 25849 ts_tablet_manager.cc:1428] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:46:09.678614 25701 heartbeater.cc:503] Master 127.7.157.126:39591 was elected leader, sending a full tablet report...
I20260929 18:46:09.678612 25850 raft_consensus.cc:348] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b86944431c4e477691c205914b18b69d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39539 } } peers { permanent_uuid: "759e8c7e73bc45348f1b0d718ac1b3d7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45495 } } peers { permanent_uuid: "a226e3c163524edd84f3b1be6f36c639" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35439 } }
I20260929 18:46:09.678860 25850 raft_consensus.cc:374] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:09.678907 25850 raft_consensus.cc:729] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a226e3c163524edd84f3b1be6f36c639, State: Initialized, Role: FOLLOWER
I20260929 18:46:09.679337 25850 consensus_queue.cc:260] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b86944431c4e477691c205914b18b69d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39539 } } peers { permanent_uuid: "759e8c7e73bc45348f1b0d718ac1b3d7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45495 } } peers { permanent_uuid: "a226e3c163524edd84f3b1be6f36c639" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35439 } }
I20260929 18:46:09.679749 25853 tablet_bootstrap.cc:654] T 2a9d69a70c1642a08c197174c2124067 P b86944431c4e477691c205914b18b69d: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:09.680284 25850 ts_tablet_manager.cc:1428] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260929 18:46:09.680665 25832 heartbeater.cc:503] Master 127.7.157.126:39591 was elected leader, sending a full tablet report...
I20260929 18:46:09.680681 25853 log.cc:824] T 2a9d69a70c1642a08c197174c2124067 P b86944431c4e477691c205914b18b69d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:09.682765 25853 tablet_bootstrap.cc:492] T 2a9d69a70c1642a08c197174c2124067 P b86944431c4e477691c205914b18b69d: No bootstrap required, opened a new log
I20260929 18:46:09.682969 25853 ts_tablet_manager.cc:1397] T 2a9d69a70c1642a08c197174c2124067 P b86944431c4e477691c205914b18b69d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:46:09.686200 25853 raft_consensus.cc:348] T 2a9d69a70c1642a08c197174c2124067 P b86944431c4e477691c205914b18b69d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b86944431c4e477691c205914b18b69d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39539 } } peers { permanent_uuid: "759e8c7e73bc45348f1b0d718ac1b3d7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45495 } } peers { permanent_uuid: "a226e3c163524edd84f3b1be6f36c639" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35439 } }
I20260929 18:46:09.686425 25853 raft_consensus.cc:374] T 2a9d69a70c1642a08c197174c2124067 P b86944431c4e477691c205914b18b69d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:09.686470 25853 raft_consensus.cc:729] T 2a9d69a70c1642a08c197174c2124067 P b86944431c4e477691c205914b18b69d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b86944431c4e477691c205914b18b69d, State: Initialized, Role: FOLLOWER
I20260929 18:46:09.686856 25853 consensus_queue.cc:260] T 2a9d69a70c1642a08c197174c2124067 P b86944431c4e477691c205914b18b69d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b86944431c4e477691c205914b18b69d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39539 } } peers { permanent_uuid: "759e8c7e73bc45348f1b0d718ac1b3d7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45495 } } peers { permanent_uuid: "a226e3c163524edd84f3b1be6f36c639" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35439 } }
I20260929 18:46:09.687497 25570 heartbeater.cc:503] Master 127.7.157.126:39591 was elected leader, sending a full tablet report...
I20260929 18:46:09.687814 25853 ts_tablet_manager.cc:1428] T 2a9d69a70c1642a08c197174c2124067 P b86944431c4e477691c205914b18b69d: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260929 18:46:09.722667 25855 raft_consensus.cc:482] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:46:09.722865 25855 raft_consensus.cc:504] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b86944431c4e477691c205914b18b69d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39539 } } peers { permanent_uuid: "759e8c7e73bc45348f1b0d718ac1b3d7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45495 } } peers { permanent_uuid: "a226e3c163524edd84f3b1be6f36c639" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35439 } }
I20260929 18:46:09.724026 25855 leader_election.cc:288] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b86944431c4e477691c205914b18b69d (127.7.157.65:39539), 759e8c7e73bc45348f1b0d718ac1b3d7 (127.7.157.66:45495)
I20260929 18:46:09.727846 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.724246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:45495 (local address 127.7.157.67:59471)
0929 18:46:09.724351 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:09.724365 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:09.724455 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:09.724750 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:09.724753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:09.724771 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:09.724895 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:09.724902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.725727 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.725731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:09.726599 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:09.726609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.727426 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.727430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:09.727528 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.727552 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:46:09.727623 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 18:46:09.727707 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":1}
I20260929 18:46:09.727872 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.724382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:59471 (local address 127.7.157.66:45495)
0929 18:46:09.724492 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:09.724508 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:09.724521 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:46:09.724580 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:09.724583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:09.724636 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:09.724730 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:09.725020 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.725621 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.726733 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.727298 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.727415 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.727465 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:09.727694 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 18:46:09.727764 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:46:09.728297 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.724367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:57479 (local address 127.7.157.65:39539)
0929 18:46:09.724486 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:09.724490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:09.724836 (+ 346us) server_negotiation.cc:407] Connection header received
0929 18:46:09.724936 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:09.724939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:09.724999 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:09.725069 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:09.725327 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.725870 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.726905 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.727508 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.727621 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.727665 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:09.728103 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 18:46:09.728182 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:46:09.728647 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.724246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:39539 (local address 127.7.157.67:57479)
0929 18:46:09.724733 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:09.724747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:09.724837 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:09.725100 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:09.725102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:09.725119 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:09.725225 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:09.725231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.725979 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.725981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:09.726774 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:09.726784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.727839 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.727845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:09.727940 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.727963 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:46:09.728437 (+ 474us) client_negotiation.cc:240] Negotiation successful
0929 18:46:09.728509 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":124,"thread_start_us":128,"threads_started":1}
I20260929 18:46:09.728605 25655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9d69a70c1642a08c197174c2124067" candidate_uuid: "a226e3c163524edd84f3b1be6f36c639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759e8c7e73bc45348f1b0d718ac1b3d7" is_pre_election: true
I20260929 18:46:09.728904 25655 raft_consensus.cc:2454] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a226e3c163524edd84f3b1be6f36c639 in term 0.
I20260929 18:46:09.729254 25524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9d69a70c1642a08c197174c2124067" candidate_uuid: "a226e3c163524edd84f3b1be6f36c639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b86944431c4e477691c205914b18b69d" is_pre_election: true
I20260929 18:46:09.729406 25720 leader_election.cc:302] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 759e8c7e73bc45348f1b0d718ac1b3d7, a226e3c163524edd84f3b1be6f36c639; no voters:
I20260929 18:46:09.729562 25524 raft_consensus.cc:2454] T 2a9d69a70c1642a08c197174c2124067 P b86944431c4e477691c205914b18b69d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a226e3c163524edd84f3b1be6f36c639 in term 0.
I20260929 18:46:09.729709 25855 raft_consensus.cc:2781] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:46:09.729795 25855 raft_consensus.cc:482] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:46:09.729843 25855 raft_consensus.cc:3037] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:09.730792 25855 raft_consensus.cc:504] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b86944431c4e477691c205914b18b69d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39539 } } peers { permanent_uuid: "759e8c7e73bc45348f1b0d718ac1b3d7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45495 } } peers { permanent_uuid: "a226e3c163524edd84f3b1be6f36c639" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35439 } }
I20260929 18:46:09.731138 25855 leader_election.cc:288] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [CANDIDATE]: Term 1 election: Requested vote from peers b86944431c4e477691c205914b18b69d (127.7.157.65:39539), 759e8c7e73bc45348f1b0d718ac1b3d7 (127.7.157.66:45495)
I20260929 18:46:09.731441 25524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9d69a70c1642a08c197174c2124067" candidate_uuid: "a226e3c163524edd84f3b1be6f36c639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b86944431c4e477691c205914b18b69d"
I20260929 18:46:09.731451 25655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a9d69a70c1642a08c197174c2124067" candidate_uuid: "a226e3c163524edd84f3b1be6f36c639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759e8c7e73bc45348f1b0d718ac1b3d7"
I20260929 18:46:09.731562 25655 raft_consensus.cc:3037] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:09.731568 25524 raft_consensus.cc:3037] T 2a9d69a70c1642a08c197174c2124067 P b86944431c4e477691c205914b18b69d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:09.732561 25655 raft_consensus.cc:2454] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a226e3c163524edd84f3b1be6f36c639 in term 1.
I20260929 18:46:09.732568 25524 raft_consensus.cc:2454] T 2a9d69a70c1642a08c197174c2124067 P b86944431c4e477691c205914b18b69d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a226e3c163524edd84f3b1be6f36c639 in term 1.
I20260929 18:46:09.732851 25722 leader_election.cc:302] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a226e3c163524edd84f3b1be6f36c639, b86944431c4e477691c205914b18b69d; no voters:
I20260929 18:46:09.733026 25855 raft_consensus.cc:2781] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:09.733225 25855 raft_consensus.cc:686] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [term 1 LEADER]: Becoming Leader. State: Replica: a226e3c163524edd84f3b1be6f36c639, State: Running, Role: LEADER
I20260929 18:46:09.733570 25855 consensus_queue.cc:237] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b86944431c4e477691c205914b18b69d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39539 } } peers { permanent_uuid: "759e8c7e73bc45348f1b0d718ac1b3d7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45495 } } peers { permanent_uuid: "a226e3c163524edd84f3b1be6f36c639" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35439 } }
W20260929 18:46:09.733819 25571 tablet.cc:2369] T 2a9d69a70c1642a08c197174c2124067 P b86944431c4e477691c205914b18b69d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:46:09.736599 25382 catalog_manager.cc:5495] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 reported cstate change: term changed from 0 to 1, leader changed from <none> to a226e3c163524edd84f3b1be6f36c639 (127.7.157.67). New cstate: current_term: 1 leader_uuid: "a226e3c163524edd84f3b1be6f36c639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b86944431c4e477691c205914b18b69d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "759e8c7e73bc45348f1b0d718ac1b3d7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a226e3c163524edd84f3b1be6f36c639" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 35439 } health_report { overall_health: HEALTHY } } }
I20260929 18:46:09.755214 25848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.752220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38070 (local address 127.7.157.67:35439)
0929 18:46:09.752325 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:09.752329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:09.752342 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:46:09.752437 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:09.752439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:09.752487 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:09.752582 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:46:09.752754 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.753231 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.753990 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:09.754190 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:09.754256 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:09.754474 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:46:09.754878 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:46:09.754962 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:09.754990 (+ 28us) server_negotiation.cc:299] Negotiation successful
0929 18:46:09.755033 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:46:09.760772 25524 raft_consensus.cc:1261] T 2a9d69a70c1642a08c197174c2124067 P b86944431c4e477691c205914b18b69d [term 1 FOLLOWER]: Refusing update from remote peer a226e3c163524edd84f3b1be6f36c639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:46:09.760771 25655 raft_consensus.cc:1261] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7 [term 1 FOLLOWER]: Refusing update from remote peer a226e3c163524edd84f3b1be6f36c639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:46:09.761444 25855 consensus_queue.cc:1035] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b86944431c4e477691c205914b18b69d" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:46:09.761638 25859 consensus_queue.cc:1035] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "759e8c7e73bc45348f1b0d718ac1b3d7" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:46:09.769609 25863 mvcc.cc:205] Tried to move back new op lower bound from 7334738205733511168 to 7334738205631467520. Current Snapshot: MvccSnapshot[applied={T|T < 7334738205733511168}]
I20260929 18:46:09.776937 25381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:46:09.777122 25381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:46:09.779526 25381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17005dc5e9a849f0a9dde571d1d94093: Sending DeleteTablet for 3 replicas of tablet 2a9d69a70c1642a08c197174c2124067
I20260929 18:46:09.780318 25635 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a9d69a70c1642a08c197174c2124067 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:09 UTC) from {username='slave'} at 127.0.0.1:57788
I20260929 18:46:09.780326 25504 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a9d69a70c1642a08c197174c2124067 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:09 UTC) from {username='slave'} at 127.0.0.1:53250
I20260929 18:46:09.780645 25766 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a9d69a70c1642a08c197174c2124067 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:09 UTC) from {username='slave'} at 127.0.0.1:38064
I20260929 18:46:09.780862 25874 tablet_replica.cc:330] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7: stopping tablet replica
I20260929 18:46:09.781064 25875 tablet_replica.cc:330] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639: stopping tablet replica
I20260929 18:46:09.781281 25874 raft_consensus.cc:2229] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:46:09.781376 25875 raft_consensus.cc:2229] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:46:09.781677 25874 raft_consensus.cc:2258] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:46:09.781757 25875 raft_consensus.cc:2258] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:46:09.781847 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 25442
I20260929 18:46:09.783463 25874 ts_tablet_manager.cc:1905] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:46:09.783571 25875 ts_tablet_manager.cc:1905] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:46:09.786240 25875 ts_tablet_manager.cc:1918] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:46:09.786302 25874 ts_tablet_manager.cc:1918] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:46:09.786398 25875 log.cc:1196] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/2a9d69a70c1642a08c197174c2124067
I20260929 18:46:09.786481 25874 log.cc:1196] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/2a9d69a70c1642a08c197174c2124067
I20260929 18:46:09.786765 25875 ts_tablet_manager.cc:1939] T 2a9d69a70c1642a08c197174c2124067 P a226e3c163524edd84f3b1be6f36c639: Deleting consensus metadata
I20260929 18:46:09.786882 25874 ts_tablet_manager.cc:1939] T 2a9d69a70c1642a08c197174c2124067 P 759e8c7e73bc45348f1b0d718ac1b3d7: Deleting consensus metadata
I20260929 18:46:09.788031 25369 catalog_manager.cc:4841] TS a226e3c163524edd84f3b1be6f36c639 (127.7.157.67:35439): tablet 2a9d69a70c1642a08c197174c2124067 (table test-table [id=45b759a338334b399f6d6c9d2ea11573]) successfully deleted
I20260929 18:46:09.788079 25367 catalog_manager.cc:4841] TS 759e8c7e73bc45348f1b0d718ac1b3d7 (127.7.157.66:45495): tablet 2a9d69a70c1642a08c197174c2124067 (table test-table [id=45b759a338334b399f6d6c9d2ea11573]) successfully deleted
W20260929 18:46:09.790318 25369 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:39539 (error 108)
I20260929 18:46:09.790577 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 25573
I20260929 18:46:09.792769 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:09.792463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:39539 (local address 127.0.0.1:53260)
0929 18:46:09.792603 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:09.792680 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:39539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 18:46:09.792925 25369 catalog_manager.cc:4584] TS b86944431c4e477691c205914b18b69d (127.7.157.65:39539): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a9d69a70c1642a08c197174c2124067: Network error: Client connection negotiation failed: client connection to 127.7.157.65:39539: connect: Connection refused (error 111)
I20260929 18:46:09.796522 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 25704
I20260929 18:46:09.801378 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 25351
2026-09-29T18:46:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1023 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:09 dist-test-slave-smpn krb5kdc[25879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:09 dist-test-slave-smpn krb5kdc[25879](info): set up 2 sockets
Sep 29 18:46:09 dist-test-slave-smpn krb5kdc[25879](info): commencing operation
krb5kdc: starting...
W20260929 18:46:10.755096 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:10 dist-test-slave-smpn krb5kdc[25879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:46:10.873723 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:41425
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:41425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:10.984088 25892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:10.984382 25892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:10.984426 25892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:10.988642 25892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:10.988708 25892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:10.988726 25892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:10.988744 25892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:10.988771 25892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:10.994210 25892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:41425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:41425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:10.995381 25892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:10.996448 25892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:11.004868 25900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:11.004882 25897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:11.004879 25898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:11.004942 25892 server_base.cc:1034] running on GCE node
I20260929 18:46:11.005580 25892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:11.006702 25892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:11.007920 25892 hybrid_clock.cc:648] HybridClock initialized: now 1790707571007890 us; error 46 us; skew 500 ppm
Sep 29 18:46:11 dist-test-slave-smpn krb5kdc[25879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:11.010673 25892 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:46:11.011632 25892 webserver.cc:460] Webserver started at http://127.7.157.126:32869/ using document root <none> and password file <none>
I20260929 18:46:11.012197 25892 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:11.012264 25892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:11.012454 25892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:11.014556 25892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "afb4ce24d41148de99e01f5a6fd9f91b"
format_stamp: "Formatted at 2026-09-29 18:46:11 on dist-test-slave-smpn"
server_key: "c5d6271dc33601b2a0e0bca92c6b10be"
server_key_iv: "f9b9e02c1eb95ed083b869f90a6d530f"
server_key_version: "encryptionkey@0"
I20260929 18:46:11.015162 25892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "afb4ce24d41148de99e01f5a6fd9f91b"
format_stamp: "Formatted at 2026-09-29 18:46:11 on dist-test-slave-smpn"
server_key: "c5d6271dc33601b2a0e0bca92c6b10be"
server_key_iv: "f9b9e02c1eb95ed083b869f90a6d530f"
server_key_version: "encryptionkey@0"
I20260929 18:46:11.018945 25892 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:46:11.021477 25907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:11.022776 25892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:46:11.022902 25892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "afb4ce24d41148de99e01f5a6fd9f91b"
format_stamp: "Formatted at 2026-09-29 18:46:11 on dist-test-slave-smpn"
server_key: "c5d6271dc33601b2a0e0bca92c6b10be"
server_key_iv: "f9b9e02c1eb95ed083b869f90a6d530f"
server_key_version: "encryptionkey@0"
I20260929 18:46:11.023013 25892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:11.038890 25892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:11.041908 25892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:11.042110 25892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:11.049052 25892 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:41425
I20260929 18:46:11.049067 25959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:41425 every 8 connection(s)
I20260929 18:46:11.050026 25892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:11.052809 25960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:11.058204 25960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b: Bootstrap starting.
I20260929 18:46:11.059631 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 25892
I20260929 18:46:11.059770 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:11.060070 7797 external_mini_cluster.cc:1428] Setting key effc0d37e91c2b988aca968306413a94
I20260929 18:46:11.061156 25960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:11.061949 25960 log.cc:824] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:11.064177 25960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b: No bootstrap required, opened a new log
I20260929 18:46:11.068464 25960 raft_consensus.cc:348] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb4ce24d41148de99e01f5a6fd9f91b" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41425 } }
I20260929 18:46:11.068785 25960 raft_consensus.cc:374] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:11.068857 25960 raft_consensus.cc:729] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb4ce24d41148de99e01f5a6fd9f91b, State: Initialized, Role: FOLLOWER
Sep 29 18:46:11 dist-test-slave-smpn krb5kdc[25879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:11.069344 25960 consensus_queue.cc:260] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb4ce24d41148de99e01f5a6fd9f91b" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41425 } }
I20260929 18:46:11.069478 25960 raft_consensus.cc:388] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:11.069550 25960 raft_consensus.cc:482] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:11.069646 25960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:11.070868 25960 raft_consensus.cc:504] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb4ce24d41148de99e01f5a6fd9f91b" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41425 } }
I20260929 18:46:11.071270 25960 leader_election.cc:302] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afb4ce24d41148de99e01f5a6fd9f91b; no voters:
I20260929 18:46:11.071575 25960 leader_election.cc:288] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:11.071659 25965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:11.071828 25965 raft_consensus.cc:686] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [term 1 LEADER]: Becoming Leader. State: Replica: afb4ce24d41148de99e01f5a6fd9f91b, State: Running, Role: LEADER
I20260929 18:46:11.072130 25965 consensus_queue.cc:237] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb4ce24d41148de99e01f5a6fd9f91b" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41425 } }
I20260929 18:46:11.072914 25960 sys_catalog.cc:564] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:11.073845 25967 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [sys.catalog]: SysCatalogTable state changed. Reason: New leader afb4ce24d41148de99e01f5a6fd9f91b. Latest consensus state: current_term: 1 leader_uuid: "afb4ce24d41148de99e01f5a6fd9f91b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb4ce24d41148de99e01f5a6fd9f91b" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41425 } } }
I20260929 18:46:11.073971 25967 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:11.074371 25966 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afb4ce24d41148de99e01f5a6fd9f91b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb4ce24d41148de99e01f5a6fd9f91b" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 41425 } } }
I20260929 18:46:11.074455 25966 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:11.074702 25970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:11.074869 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:11.061259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51288 (local address 127.7.157.126:41425)
0929 18:46:11.061679 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:11.061696 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:11.061732 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:46:11.062489 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:11.062528 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:11.062928 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:11.063263 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:11.064089 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.065128 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.066305 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.066806 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.066927 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:11.069793 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:11.069799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:11.069814 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:11.069843 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:11.071584 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:11.072082 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:11.072088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:11.072095 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:11.072240 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:11.072511 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:11.072514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:11.072516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:11.072978 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:11.073176 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:11.073472 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 18:46:11.073806 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":115,"threads_started":1}
I20260929 18:46:11.078084 25970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:11.082957 25970 catalog_manager.cc:1347] Generated new cluster ID: a68e22196d79464eaf5667c40e38530e
I20260929 18:46:11.083050 25970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:11.094585 25970 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:11.095587 25970 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:11.102650 25970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b: Generated new TSK 0
I20260929 18:46:11.103394 25970 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:46:11.134155 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:41425
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:11.244040 25986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:11.244302 25986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:11.244347 25986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:11.248374 25986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:11.248443 25986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:11.248517 25986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:46:11.253926 25986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:41425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:11.255097 25986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:11.256161 25986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:11.265863 25994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:11.265852 25992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:11.265849 25991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:11.265936 25986 server_base.cc:1034] running on GCE node
I20260929 18:46:11.266484 25986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:11.267203 25986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:11.268352 25986 hybrid_clock.cc:648] HybridClock initialized: now 1790707571268315 us; error 51 us; skew 500 ppm
Sep 29 18:46:11 dist-test-slave-smpn krb5kdc[25879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:11.270952 25986 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:46:11.271829 25986 webserver.cc:460] Webserver started at http://127.7.157.65:39709/ using document root <none> and password file <none>
I20260929 18:46:11.272377 25986 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:11.272441 25986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:11.272629 25986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:11.274734 25986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "3bf7ca5de63d46719546b8cbc80fbd12"
format_stamp: "Formatted at 2026-09-29 18:46:11 on dist-test-slave-smpn"
server_key: "163d2a6292cab6849ce61442f3f1abcc"
server_key_iv: "55e7913d4613b442142da599d26caade"
server_key_version: "encryptionkey@0"
I20260929 18:46:11.275278 25986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "3bf7ca5de63d46719546b8cbc80fbd12"
format_stamp: "Formatted at 2026-09-29 18:46:11 on dist-test-slave-smpn"
server_key: "163d2a6292cab6849ce61442f3f1abcc"
server_key_iv: "55e7913d4613b442142da599d26caade"
server_key_version: "encryptionkey@0"
I20260929 18:46:11.278867 25986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:46:11.281258 26001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:11.282423 25986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:46:11.282526 25986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "3bf7ca5de63d46719546b8cbc80fbd12"
format_stamp: "Formatted at 2026-09-29 18:46:11 on dist-test-slave-smpn"
server_key: "163d2a6292cab6849ce61442f3f1abcc"
server_key_iv: "55e7913d4613b442142da599d26caade"
server_key_version: "encryptionkey@0"
I20260929 18:46:11.282616 25986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:11.301153 25986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:11.303897 25986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:11.304082 25986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:11.304682 25986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:11.305503 25986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:11.305548 25986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:11.305600 25986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:11.305629 25986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:11.313774 25986 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:36491
I20260929 18:46:11.313807 26114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:36491 every 8 connection(s)
I20260929 18:46:11.314641 25986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:46:11.320272 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 25986
I20260929 18:46:11.320396 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:46:11.320674 7797 external_mini_cluster.cc:1428] Setting key 3c170048b8e09caeb6cc3e68d9db81e6
Sep 29 18:46:11 dist-test-slave-smpn krb5kdc[25879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:11.328830 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:11.316404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:42481 (local address 127.7.157.126:41425)
0929 18:46:11.316542 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:11.316547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:11.317420 (+ 873us) server_negotiation.cc:407] Connection header received
0929 18:46:11.318892 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:11.318894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:11.318941 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:11.319019 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:11.320291 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.320909 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.322055 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.322288 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.322367 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:11.325105 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:11.325109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:11.325111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:11.325139 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:11.326643 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:11.327161 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:11.327165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:11.327167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:11.327265 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:11.327642 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:11.327645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:11.327647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:11.327892 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:11.328012 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:11.328510 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 18:46:11.328655 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:46:11.329649 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:11.316834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41425 (local address 127.7.157.65:42481)
0929 18:46:11.317266 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:11.317312 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:11.318692 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:11.319213 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:11.319226 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:11.319597 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:11.320147 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:11.320168 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.321023 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.321027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:11.321939 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:11.321948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.322444 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.322449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:11.322658 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:11.323291 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:11.323310 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:11.324923 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:11.326754 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:11.326758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:11.326773 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:11.327052 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:11.327369 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:11.327371 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:11.327373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:11.327534 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:11.327992 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:11.327997 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:11.328264 (+ 267us) client_negotiation.cc:769] Sending connection context
0929 18:46:11.328419 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 18:46:11.328709 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
I20260929 18:46:11.330883 26115 heartbeater.cc:347] Connected to a master server at 127.7.157.126:41425
I20260929 18:46:11.331200 26115 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:11.331727 26115 heartbeater.cc:511] Master 127.7.157.126:41425 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
I20260929 18:46:11.333325 25924 ts_manager.cc:194] Registered new tserver with Master: 3bf7ca5de63d46719546b8cbc80fbd12 (127.7.157.65:36491)
I20260929 18:46:11.334460 25924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:42481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:46:11.346343 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:41425
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:11.456082 26120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:11.456333 26120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:11.456372 26120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:11.460261 26120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:11.460311 26120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:11.460366 26120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:46:11.465607 26120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:41425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:11.466539 26120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:11.467523 26120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:11.477006 26125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:11.477110 26128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:11.477062 26126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:11.477150 26120 server_base.cc:1034] running on GCE node
I20260929 18:46:11.477592 26120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:11.478226 26120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:11.479373 26120 hybrid_clock.cc:648] HybridClock initialized: now 1790707571479351 us; error 34 us; skew 500 ppm
Sep 29 18:46:11 dist-test-slave-smpn krb5kdc[25879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:11.485194 26120 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:46:11.486142 26120 webserver.cc:460] Webserver started at http://127.7.157.66:38003/ using document root <none> and password file <none>
I20260929 18:46:11.486739 26120 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:11.486811 26120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:11.487020 26120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:11.489200 26120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "f092db8730424004a7eda1df8344ce49"
format_stamp: "Formatted at 2026-09-29 18:46:11 on dist-test-slave-smpn"
server_key: "f83dd4a67225175334c5ffeef2da062d"
server_key_iv: "b77d3cd42b6f2e0898b5db3c780e24ef"
server_key_version: "encryptionkey@0"
I20260929 18:46:11.489708 26120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "f092db8730424004a7eda1df8344ce49"
format_stamp: "Formatted at 2026-09-29 18:46:11 on dist-test-slave-smpn"
server_key: "f83dd4a67225175334c5ffeef2da062d"
server_key_iv: "b77d3cd42b6f2e0898b5db3c780e24ef"
server_key_version: "encryptionkey@0"
I20260929 18:46:11.493258 26120 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:46:11.495508 26135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:11.496616 26120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:11.496728 26120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "f092db8730424004a7eda1df8344ce49"
format_stamp: "Formatted at 2026-09-29 18:46:11 on dist-test-slave-smpn"
server_key: "f83dd4a67225175334c5ffeef2da062d"
server_key_iv: "b77d3cd42b6f2e0898b5db3c780e24ef"
server_key_version: "encryptionkey@0"
I20260929 18:46:11.496817 26120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:11.524168 26120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:11.527058 26120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:11.527259 26120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:11.527868 26120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:11.528704 26120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:11.528751 26120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:11.528802 26120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:11.528831 26120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:11.537068 26120 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:43185
I20260929 18:46:11.537096 26248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:43185 every 8 connection(s)
I20260929 18:46:11.537961 26120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:46:11.542899 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 26120
I20260929 18:46:11.543013 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:46:11.543323 7797 external_mini_cluster.cc:1428] Setting key d217fe8c580f3d791eefd5c4d8f02c07
Sep 29 18:46:11 dist-test-slave-smpn krb5kdc[25879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:11.551473 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:11.539656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:40057 (local address 127.7.157.126:41425)
0929 18:46:11.539823 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:11.539829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:11.540718 (+ 889us) server_negotiation.cc:407] Connection header received
0929 18:46:11.541890 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:11.541893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:11.541938 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:11.542013 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:11.543252 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.543935 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.545215 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.545435 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.545519 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:11.548047 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:11.548051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:11.548054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:11.548079 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:11.549543 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:11.550008 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:11.550012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:11.550014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:11.550105 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:11.550386 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:11.550390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:11.550392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:11.550611 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:11.550709 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:11.551185 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 18:46:11.551307 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:46:11.552237 26251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:11.540093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41425 (local address 127.7.157.66:40057)
0929 18:46:11.540564 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:11.540609 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:11.541702 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:11.542168 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:11.542174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:11.542538 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:11.543105 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:11.543125 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.544067 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.544070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:11.545098 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:11.545107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.545570 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.545574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:11.545731 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:11.546287 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:11.546306 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:11.547879 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:11.549651 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:11.549656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:11.549668 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:11.549911 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:11.550184 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:11.550186 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:11.550187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:11.550306 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:11.550694 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:11.550699 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:11.550929 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 18:46:11.551126 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 18:46:11.551383 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
I20260929 18:46:11.553251 26249 heartbeater.cc:347] Connected to a master server at 127.7.157.126:41425
I20260929 18:46:11.553472 26249 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:11.553933 26249 heartbeater.cc:511] Master 127.7.157.126:41425 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
I20260929 18:46:11.554914 25924 ts_manager.cc:194] Registered new tserver with Master: f092db8730424004a7eda1df8344ce49 (127.7.157.66:43185)
I20260929 18:46:11.555629 25924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:40057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:46:11.569521 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:41425
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:11.676803 26254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:11.677045 26254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:11.677094 26254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:11.681102 26254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:11.681162 26254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:11.681233 26254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:46:11.686489 26254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:41425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:11.687604 26254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:11.688659 26254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:11.697856 26260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:11.697839 26259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:11.697858 26262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:11.698024 26254 server_base.cc:1034] running on GCE node
I20260929 18:46:11.698519 26254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:11.699172 26254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:11.700354 26254 hybrid_clock.cc:648] HybridClock initialized: now 1790707571700326 us; error 37 us; skew 500 ppm
Sep 29 18:46:11 dist-test-slave-smpn krb5kdc[25879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:11.703018 26254 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:46:11.703861 26254 webserver.cc:460] Webserver started at http://127.7.157.67:36471/ using document root <none> and password file <none>
I20260929 18:46:11.704433 26254 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:11.704499 26254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:11.704679 26254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:11.706689 26254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "45efceb6c9fd442c9d28096853f2098e"
format_stamp: "Formatted at 2026-09-29 18:46:11 on dist-test-slave-smpn"
server_key: "ce3440261545ee65437f20fa146a2098"
server_key_iv: "05dd7843547e6116116dd2f29b365f87"
server_key_version: "encryptionkey@0"
I20260929 18:46:11.707248 26254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "45efceb6c9fd442c9d28096853f2098e"
format_stamp: "Formatted at 2026-09-29 18:46:11 on dist-test-slave-smpn"
server_key: "ce3440261545ee65437f20fa146a2098"
server_key_iv: "05dd7843547e6116116dd2f29b365f87"
server_key_version: "encryptionkey@0"
I20260929 18:46:11.710860 26254 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260929 18:46:11.713366 26269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:11.714666 26254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:46:11.714777 26254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "45efceb6c9fd442c9d28096853f2098e"
format_stamp: "Formatted at 2026-09-29 18:46:11 on dist-test-slave-smpn"
server_key: "ce3440261545ee65437f20fa146a2098"
server_key_iv: "05dd7843547e6116116dd2f29b365f87"
server_key_version: "encryptionkey@0"
I20260929 18:46:11.714876 26254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:11.734819 26254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:11.738149 26254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:11.738335 26254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:11.738936 26254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:11.739815 26254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:11.739876 26254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:11.739931 26254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:11.739956 26254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:11.748428 26254 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:37891
I20260929 18:46:11.748458 26382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:37891 every 8 connection(s)
I20260929 18:46:11.749339 26254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:46:11.755102 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 26254
I20260929 18:46:11.755223 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:46:11.755507 7797 external_mini_cluster.cc:1428] Setting key e41e6a0c3f6fc44f69550ad03e400ab2
Sep 29 18:46:11 dist-test-slave-smpn krb5kdc[25879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:11.762877 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:11.751107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:34353 (local address 127.7.157.126:41425)
0929 18:46:11.751227 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:11.751231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:11.752095 (+ 864us) server_negotiation.cc:407] Connection header received
0929 18:46:11.753465 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:11.753469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:11.753529 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:11.753608 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:11.755092 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.755874 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.757252 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.757510 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.757601 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:11.759935 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:11.759938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:11.759941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:11.759966 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:11.761244 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:11.761680 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:11.761682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:11.761683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:11.761749 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:11.762019 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:11.762021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:11.762023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:11.762185 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:11.762254 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:11.762665 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 18:46:11.762759 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:46:11.763775 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:11.751512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:41425 (local address 127.7.157.67:34353)
0929 18:46:11.751943 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:11.751990 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:11.753241 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:11.753788 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:11.753795 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:11.754252 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:11.754918 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:11.754935 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.756003 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.756007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:11.757143 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:11.757152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.757646 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.757650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:11.757828 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:11.758308 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:11.758324 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:11.759780 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:11.761343 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:11.761347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:11.761359 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:11.761608 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:11.761809 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:11.761811 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:11.761812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:11.761933 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:11.762253 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:11.762257 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:11.762482 (+ 225us) client_negotiation.cc:769] Sending connection context
0929 18:46:11.762636 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 18:46:11.762887 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":100,"threads_started":1}
I20260929 18:46:11.764849 26383 heartbeater.cc:347] Connected to a master server at 127.7.157.126:41425
I20260929 18:46:11.765071 26383 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:11.765539 26383 heartbeater.cc:511] Master 127.7.157.126:41425 requested a full tablet report, sending...
I20260929 18:46:11.766479 25924 ts_manager.cc:194] Registered new tserver with Master: 45efceb6c9fd442c9d28096853f2098e (127.7.157.67:37891)
I20260929 18:46:11.767025 25924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:34353
I20260929 18:46:11.769196 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:46:11.777680 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:11.770965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51302 (local address 127.7.157.126:41425)
0929 18:46:11.771108 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:11.771112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:11.771287 (+ 175us) server_negotiation.cc:407] Connection header received
0929 18:46:11.771391 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:11.771394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:11.771430 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:11.771497 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:11.772075 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.772565 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.773377 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.773595 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.773662 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:11.775144 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:11.775147 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:11.775149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:11.775168 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:11.776267 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:11.776681 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:11.776683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:11.776684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:11.776750 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:11.776970 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:11.776973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:11.776975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:11.777154 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:11.777261 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:11.777435 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 18:46:11.777524 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:46:11.780431 25924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:46:11.783452 25924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:46:11.797989 26394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:11.791925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:37891 (local address 127.0.0.1:44934)
0929 18:46:11.792293 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:11.792347 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:11.792479 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:11.793614 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:11.793620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:11.793882 (+ 262us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:11.794039 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:11.794046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.795538 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.795541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:11.796499 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:11.796509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.797409 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.797412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:11.797538 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:11.797577 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 18:46:11.797684 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 18:46:11.797816 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":75,"thread_start_us":98,"threads_started":1}
I20260929 18:46:11.798131 26399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:11.793087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.7.157.66:43185)
0929 18:46:11.793382 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:11.793394 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:11.793416 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:46:11.793484 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:11.793488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:11.793669 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:11.794026 (+ 357us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:11.794175 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.795009 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.796652 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.797500 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.797654 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:11.797810 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:11.797863 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 18:46:11.797963 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":104,"threads_started":1}
I20260929 18:46:11.798381 26395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:11.792036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44934 (local address 127.7.157.67:37891)
0929 18:46:11.792719 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:11.792734 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:11.792759 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:46:11.793021 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:11.793028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:11.793267 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:11.793399 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:11.794200 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.795362 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.796627 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.797297 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.797397 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:11.797557 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:11.797991 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 18:46:11.798247 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":171,"threads_started":1}
I20260929 18:46:11.798607 26396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:11.792337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:36491 (local address 127.0.0.1:40732)
0929 18:46:11.792812 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:11.792825 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:11.792917 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:11.793623 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:11.793626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:11.793776 (+ 150us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:11.794021 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:11.794031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.795514 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.795517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:11.796716 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:11.796724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.797738 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.797741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:11.798087 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:11.798343 (+ 256us) client_negotiation.cc:769] Sending connection context
0929 18:46:11.798403 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:46:11.798471 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":378,"spinlock_wait_cycles":44928,"thread_start_us":83,"threads_started":1}
I20260929 18:46:11.798699 26397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:11.792579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.7.157.65:36491)
0929 18:46:11.793174 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:11.793188 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:11.793213 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:46:11.793285 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:11.793289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:11.793476 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:11.793699 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:11.794148 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.795362 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.796835 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.797620 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.797786 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:11.797909 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:11.798462 (+ 553us) server_negotiation.cc:299] Negotiation successful
0929 18:46:11.798567 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":79,"threads_started":1}
I20260929 18:46:11.797996 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:11.792690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:43185 (local address 127.0.0.1:50436)
0929 18:46:11.793054 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:11.793067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:11.793142 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:11.793810 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:11.793813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:11.793827 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:11.794021 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:11.794031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.795141 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.795144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:11.796515 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:11.796526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:11.797651 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:11.797654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:11.797733 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:11.797770 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 18:46:11.797820 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 18:46:11.797874 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":85,"thread_start_us":87,"threads_started":1}
I20260929 18:46:11.801476 26049 tablet_service.cc:1467] Processing CreateTablet for tablet 174c80408fdb4452a48214262c19e522 (DEFAULT_TABLE table=test-table [id=dc120201922e45cbb0f8f2e3bd71aa74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:11.801528 26317 tablet_service.cc:1467] Processing CreateTablet for tablet 174c80408fdb4452a48214262c19e522 (DEFAULT_TABLE table=test-table [id=dc120201922e45cbb0f8f2e3bd71aa74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:11.802477 26049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174c80408fdb4452a48214262c19e522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:11.802556 26317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174c80408fdb4452a48214262c19e522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:11.802420 26183 tablet_service.cc:1467] Processing CreateTablet for tablet 174c80408fdb4452a48214262c19e522 (DEFAULT_TABLE table=test-table [id=dc120201922e45cbb0f8f2e3bd71aa74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:11.803381 26183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174c80408fdb4452a48214262c19e522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:11.809667 26400 tablet_bootstrap.cc:492] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12: Bootstrap starting.
I20260929 18:46:11.810372 26402 tablet_bootstrap.cc:492] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49: Bootstrap starting.
I20260929 18:46:11.811426 26401 tablet_bootstrap.cc:492] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e: Bootstrap starting.
I20260929 18:46:11.811875 26400 tablet_bootstrap.cc:654] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:11.812829 26402 tablet_bootstrap.cc:654] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:11.812934 26400 log.cc:824] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:11.813560 26401 tablet_bootstrap.cc:654] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:11.813632 26402 log.cc:824] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:11.814658 26401 log.cc:824] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:11.815033 26400 tablet_bootstrap.cc:492] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12: No bootstrap required, opened a new log
I20260929 18:46:11.815307 26400 ts_tablet_manager.cc:1397] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260929 18:46:11.815842 26402 tablet_bootstrap.cc:492] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49: No bootstrap required, opened a new log
I20260929 18:46:11.816040 26402 ts_tablet_manager.cc:1397] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260929 18:46:11.816442 26401 tablet_bootstrap.cc:492] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e: No bootstrap required, opened a new log
I20260929 18:46:11.816612 26401 ts_tablet_manager.cc:1397] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:46:11.819312 26400 raft_consensus.cc:348] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45efceb6c9fd442c9d28096853f2098e" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37891 } } peers { permanent_uuid: "f092db8730424004a7eda1df8344ce49" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 43185 } } peers { permanent_uuid: "3bf7ca5de63d46719546b8cbc80fbd12" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36491 } }
I20260929 18:46:11.819620 26400 raft_consensus.cc:374] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:11.819679 26400 raft_consensus.cc:729] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bf7ca5de63d46719546b8cbc80fbd12, State: Initialized, Role: FOLLOWER
I20260929 18:46:11.819546 26401 raft_consensus.cc:348] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45efceb6c9fd442c9d28096853f2098e" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37891 } } peers { permanent_uuid: "f092db8730424004a7eda1df8344ce49" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 43185 } } peers { permanent_uuid: "3bf7ca5de63d46719546b8cbc80fbd12" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36491 } }
I20260929 18:46:11.819804 26401 raft_consensus.cc:374] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:11.819859 26401 raft_consensus.cc:729] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45efceb6c9fd442c9d28096853f2098e, State: Initialized, Role: FOLLOWER
I20260929 18:46:11.819725 26402 raft_consensus.cc:348] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45efceb6c9fd442c9d28096853f2098e" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37891 } } peers { permanent_uuid: "f092db8730424004a7eda1df8344ce49" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 43185 } } peers { permanent_uuid: "3bf7ca5de63d46719546b8cbc80fbd12" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36491 } }
I20260929 18:46:11.820045 26402 raft_consensus.cc:374] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:11.820093 26402 raft_consensus.cc:729] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f092db8730424004a7eda1df8344ce49, State: Initialized, Role: FOLLOWER
I20260929 18:46:11.820151 26400 consensus_queue.cc:260] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45efceb6c9fd442c9d28096853f2098e" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37891 } } peers { permanent_uuid: "f092db8730424004a7eda1df8344ce49" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 43185 } } peers { permanent_uuid: "3bf7ca5de63d46719546b8cbc80fbd12" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36491 } }
I20260929 18:46:11.820264 26401 consensus_queue.cc:260] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45efceb6c9fd442c9d28096853f2098e" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37891 } } peers { permanent_uuid: "f092db8730424004a7eda1df8344ce49" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 43185 } } peers { permanent_uuid: "3bf7ca5de63d46719546b8cbc80fbd12" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36491 } }
I20260929 18:46:11.820463 26402 consensus_queue.cc:260] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45efceb6c9fd442c9d28096853f2098e" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37891 } } peers { permanent_uuid: "f092db8730424004a7eda1df8344ce49" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 43185 } } peers { permanent_uuid: "3bf7ca5de63d46719546b8cbc80fbd12" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36491 } }
I20260929 18:46:11.820974 26115 heartbeater.cc:503] Master 127.7.157.126:41425 was elected leader, sending a full tablet report...
I20260929 18:46:11.821240 26401 ts_tablet_manager.cc:1428] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:46:11.821260 26402 ts_tablet_manager.cc:1428] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260929 18:46:11.821251 26400 ts_tablet_manager.cc:1428] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260929 18:46:11.821576 26249 heartbeater.cc:503] Master 127.7.157.126:41425 was elected leader, sending a full tablet report...
I20260929 18:46:11.822165 26383 heartbeater.cc:503] Master 127.7.157.126:41425 was elected leader, sending a full tablet report...
W20260929 18:46:12.000274 26384 tablet.cc:2369] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:46:12.039122 26250 tablet.cc:2369] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:46:12.042917 26406 raft_consensus.cc:482] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:46:12.043143 26406 raft_consensus.cc:504] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45efceb6c9fd442c9d28096853f2098e" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37891 } } peers { permanent_uuid: "f092db8730424004a7eda1df8344ce49" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 43185 } } peers { permanent_uuid: "3bf7ca5de63d46719546b8cbc80fbd12" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36491 } }
I20260929 18:46:12.044353 26406 leader_election.cc:288] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45efceb6c9fd442c9d28096853f2098e (127.7.157.67:37891), f092db8730424004a7eda1df8344ce49 (127.7.157.66:43185)
I20260929 18:46:12.048573 26399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.045165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:34753 (local address 127.7.157.66:43185)
0929 18:46:12.045311 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:12.045315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:12.045491 (+ 176us) server_negotiation.cc:407] Connection header received
0929 18:46:12.045583 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:12.045585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:12.045628 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:12.045689 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:12.045958 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.046527 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.047615 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.048000 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.048108 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:12.048161 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:12.048379 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 18:46:12.048446 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:46:12.048574 26395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.044740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:51679 (local address 127.7.157.67:37891)
0929 18:46:12.044895 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:12.044900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:12.044944 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:46:12.045143 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:12.045146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:12.045200 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:12.045288 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:12.045580 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.046218 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.047522 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.047968 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.048070 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:12.048126 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:12.048355 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 18:46:12.048429 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 18:46:12.048590 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.044628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:37891 (local address 127.7.157.65:51679)
0929 18:46:12.044869 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:12.044886 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:12.045005 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:12.045308 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:12.045312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:12.045336 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:12.045455 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:12.045461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.046331 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.046334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:12.047397 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:12.047409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.048087 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.048091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:12.048188 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:12.048263 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 18:46:12.048321 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:46:12.048397 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":25,"thread_start_us":72,"threads_started":1}
I20260929 18:46:12.048591 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.045032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:43185 (local address 127.7.157.65:34753)
0929 18:46:12.045406 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:12.045419 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:12.045502 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:12.045718 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:12.045721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:12.045737 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:12.045851 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:12.045857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.046639 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.046641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:12.047497 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:12.047508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.048113 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.048117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:12.048265 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:12.048283 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:46:12.048352 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:46:12.048423 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":22,"thread_start_us":105,"threads_started":1}
I20260929 18:46:12.049481 26203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174c80408fdb4452a48214262c19e522" candidate_uuid: "3bf7ca5de63d46719546b8cbc80fbd12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f092db8730424004a7eda1df8344ce49" is_pre_election: true
I20260929 18:46:12.049481 26337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174c80408fdb4452a48214262c19e522" candidate_uuid: "3bf7ca5de63d46719546b8cbc80fbd12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45efceb6c9fd442c9d28096853f2098e" is_pre_election: true
I20260929 18:46:12.049832 26203 raft_consensus.cc:2454] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bf7ca5de63d46719546b8cbc80fbd12 in term 0.
I20260929 18:46:12.049832 26337 raft_consensus.cc:2454] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bf7ca5de63d46719546b8cbc80fbd12 in term 0.
I20260929 18:46:12.050333 26004 leader_election.cc:302] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bf7ca5de63d46719546b8cbc80fbd12, f092db8730424004a7eda1df8344ce49; no voters:
I20260929 18:46:12.050623 26406 raft_consensus.cc:2781] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:46:12.050724 26406 raft_consensus.cc:482] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:46:12.050765 26406 raft_consensus.cc:3037] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:12.052100 26406 raft_consensus.cc:504] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45efceb6c9fd442c9d28096853f2098e" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37891 } } peers { permanent_uuid: "f092db8730424004a7eda1df8344ce49" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 43185 } } peers { permanent_uuid: "3bf7ca5de63d46719546b8cbc80fbd12" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36491 } }
I20260929 18:46:12.052476 26406 leader_election.cc:288] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [CANDIDATE]: Term 1 election: Requested vote from peers 45efceb6c9fd442c9d28096853f2098e (127.7.157.67:37891), f092db8730424004a7eda1df8344ce49 (127.7.157.66:43185)
I20260929 18:46:12.052882 26337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174c80408fdb4452a48214262c19e522" candidate_uuid: "3bf7ca5de63d46719546b8cbc80fbd12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45efceb6c9fd442c9d28096853f2098e"
I20260929 18:46:12.053022 26337 raft_consensus.cc:3037] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:12.053081 26203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174c80408fdb4452a48214262c19e522" candidate_uuid: "3bf7ca5de63d46719546b8cbc80fbd12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f092db8730424004a7eda1df8344ce49"
I20260929 18:46:12.053200 26203 raft_consensus.cc:3037] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:12.054337 26337 raft_consensus.cc:2454] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bf7ca5de63d46719546b8cbc80fbd12 in term 1.
I20260929 18:46:12.054337 26203 raft_consensus.cc:2454] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bf7ca5de63d46719546b8cbc80fbd12 in term 1.
I20260929 18:46:12.054653 26004 leader_election.cc:302] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bf7ca5de63d46719546b8cbc80fbd12, 45efceb6c9fd442c9d28096853f2098e; no voters:
I20260929 18:46:12.054811 26406 raft_consensus.cc:2781] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:12.055013 26406 raft_consensus.cc:686] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [term 1 LEADER]: Becoming Leader. State: Replica: 3bf7ca5de63d46719546b8cbc80fbd12, State: Running, Role: LEADER
I20260929 18:46:12.055368 26406 consensus_queue.cc:237] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45efceb6c9fd442c9d28096853f2098e" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37891 } } peers { permanent_uuid: "f092db8730424004a7eda1df8344ce49" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 43185 } } peers { permanent_uuid: "3bf7ca5de63d46719546b8cbc80fbd12" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36491 } }
I20260929 18:46:12.058717 25923 catalog_manager.cc:5495] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bf7ca5de63d46719546b8cbc80fbd12 (127.7.157.65). New cstate: current_term: 1 leader_uuid: "3bf7ca5de63d46719546b8cbc80fbd12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45efceb6c9fd442c9d28096853f2098e" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f092db8730424004a7eda1df8344ce49" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 43185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bf7ca5de63d46719546b8cbc80fbd12" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 36491 } health_report { overall_health: HEALTHY } } }
W20260929 18:46:12.065939 26116 tablet.cc:2369] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:46:12.091483 26397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.088199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40742 (local address 127.7.157.65:36491)
0929 18:46:12.088372 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:12.088377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:12.088395 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:46:12.088478 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:12.088481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:12.088545 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:12.088637 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:46:12.088830 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.089341 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.090140 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.090345 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.090416 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:12.090651 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:46:12.091108 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:46:12.091233 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:12.091292 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 18:46:12.091333 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 18:46:12.097263 26337 raft_consensus.cc:1261] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e [term 1 FOLLOWER]: Refusing update from remote peer 3bf7ca5de63d46719546b8cbc80fbd12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:46:12.097265 26203 raft_consensus.cc:1261] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49 [term 1 FOLLOWER]: Refusing update from remote peer 3bf7ca5de63d46719546b8cbc80fbd12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:46:12.097935 26406 consensus_queue.cc:1035] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f092db8730424004a7eda1df8344ce49" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 43185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:46:12.098140 26411 consensus_queue.cc:1035] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45efceb6c9fd442c9d28096853f2098e" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:46:12.106449 26415 mvcc.cc:205] Tried to move back new op lower bound from 7334738215303413760 to 7334738215141580800. Current Snapshot: MvccSnapshot[applied={T|T < 7334738215303413760}]
I20260929 18:46:12.113823 25924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:46:12.113986 25924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:46:12.116240 25924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P afb4ce24d41148de99e01f5a6fd9f91b: Sending DeleteTablet for 3 replicas of tablet 174c80408fdb4452a48214262c19e522
I20260929 18:46:12.116981 26049 tablet_service.cc:1514] Processing DeleteTablet for tablet 174c80408fdb4452a48214262c19e522 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:12 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:40732
I20260929 18:46:12.116976 26183 tablet_service.cc:1514] Processing DeleteTablet for tablet 174c80408fdb4452a48214262c19e522 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:12 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:50436
I20260929 18:46:12.117506 26424 tablet_replica.cc:330] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12: stopping tablet replica
I20260929 18:46:12.117529 26317 tablet_service.cc:1514] Processing DeleteTablet for tablet 174c80408fdb4452a48214262c19e522 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:12 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:44934
I20260929 18:46:12.117991 26426 tablet_replica.cc:330] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e: stopping tablet replica
I20260929 18:46:12.118086 26425 tablet_replica.cc:330] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49: stopping tablet replica
I20260929 18:46:12.118366 26426 raft_consensus.cc:2229] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:46:12.118382 26425 raft_consensus.cc:2229] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:46:12.118595 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 25986
I20260929 18:46:12.118719 26425 raft_consensus.cc:2258] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:46:12.117841 26424 raft_consensus.cc:2229] T 174c80408fdb4452a48214262c19e522 P 3bf7ca5de63d46719546b8cbc80fbd12 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:46:12.120013 26426 raft_consensus.cc:2258] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:46:12.121047 26425 ts_tablet_manager.cc:1905] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:46:12.121340 26426 ts_tablet_manager.cc:1905] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:46:12.123502 26426 ts_tablet_manager.cc:1918] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:46:12.123591 26426 log.cc:1196] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/174c80408fdb4452a48214262c19e522
I20260929 18:46:12.123783 26425 ts_tablet_manager.cc:1918] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:46:12.123857 26426 ts_tablet_manager.cc:1939] T 174c80408fdb4452a48214262c19e522 P 45efceb6c9fd442c9d28096853f2098e: Deleting consensus metadata
I20260929 18:46:12.123875 26425 log.cc:1196] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/174c80408fdb4452a48214262c19e522
I20260929 18:46:12.124147 26425 ts_tablet_manager.cc:1939] T 174c80408fdb4452a48214262c19e522 P f092db8730424004a7eda1df8344ce49: Deleting consensus metadata
I20260929 18:46:12.125131 25910 catalog_manager.cc:4841] TS 45efceb6c9fd442c9d28096853f2098e (127.7.157.67:37891): tablet 174c80408fdb4452a48214262c19e522 (table test-table [id=dc120201922e45cbb0f8f2e3bd71aa74]) successfully deleted
I20260929 18:46:12.125624 25910 catalog_manager.cc:4841] TS f092db8730424004a7eda1df8344ce49 (127.7.157.66:43185): tablet 174c80408fdb4452a48214262c19e522 (table test-table [id=dc120201922e45cbb0f8f2e3bd71aa74]) successfully deleted
I20260929 18:46:12.126336 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 26120
W20260929 18:46:12.126384 25909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:36491 (error 108)
I20260929 18:46:12.128544 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.128282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:36491 (local address 127.0.0.1:40744)
0929 18:46:12.128406 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:12.128470 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:36491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 18:46:12.128684 25909 catalog_manager.cc:4584] TS 3bf7ca5de63d46719546b8cbc80fbd12 (127.7.157.65:36491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 174c80408fdb4452a48214262c19e522: Network error: Client connection negotiation failed: client connection to 127.7.157.65:36491: connect: Connection refused (error 111)
I20260929 18:46:12.131321 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 26254
I20260929 18:46:12.136282 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 25892
2026-09-29T18:46:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2335 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T18:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:12Z Disabled control of system clock
I20260929 18:46:12.164206 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:35653
--webserver_interface=127.7.157.126
--webserver_port=0
--builtin_ntp_servers=127.7.157.84:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:35653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:46:12.273049 26431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:12.273320 26431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:12.273355 26431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:12.277312 26431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:12.277370 26431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:12.277385 26431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:12.277400 26431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:12.277415 26431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:12.282194 26431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:35653
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:35653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:12.283308 26431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:12.284703 26431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:12.293288 26436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:12.293304 26437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:12.293344 26431 server_base.cc:1034] running on GCE node
W20260929 18:46:12.293529 26439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:12.294096 26431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:12.295497 26431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:12.296671 26431 hybrid_clock.cc:648] HybridClock initialized: now 1790707572296627 us; error 55 us; skew 500 ppm
I20260929 18:46:12.298925 26431 webserver.cc:460] Webserver started at http://127.7.157.126:44389/ using document root <none> and password file <none>
I20260929 18:46:12.299523 26431 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:12.299584 26431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:12.299749 26431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:12.301762 26431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "b5a9a5de734847b9a7a9c5d128c8ce43"
format_stamp: "Formatted at 2026-09-29 18:46:12 on dist-test-slave-smpn"
server_key: "c0449fab95dfb453548f08c2f44fd9a5"
server_key_iv: "a5005534666f395d358d0e77273f4a52"
server_key_version: "encryptionkey@0"
I20260929 18:46:12.302270 26431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "b5a9a5de734847b9a7a9c5d128c8ce43"
format_stamp: "Formatted at 2026-09-29 18:46:12 on dist-test-slave-smpn"
server_key: "c0449fab95dfb453548f08c2f44fd9a5"
server_key_iv: "a5005534666f395d358d0e77273f4a52"
server_key_version: "encryptionkey@0"
I20260929 18:46:12.305824 26431 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:46:12.308220 26445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:12.309446 26431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:46:12.309548 26431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "b5a9a5de734847b9a7a9c5d128c8ce43"
format_stamp: "Formatted at 2026-09-29 18:46:12 on dist-test-slave-smpn"
server_key: "c0449fab95dfb453548f08c2f44fd9a5"
server_key_iv: "a5005534666f395d358d0e77273f4a52"
server_key_version: "encryptionkey@0"
I20260929 18:46:12.309634 26431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:12.332847 26431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:12.333560 26431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:12.333719 26431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:12.340535 26431 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:35653
I20260929 18:46:12.340550 26497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:35653 every 8 connection(s)
I20260929 18:46:12.341512 26431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:12.344545 26498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:12.349362 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 26431
I20260929 18:46:12.349504 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:12.349786 7797 external_mini_cluster.cc:1428] Setting key ea6eb581bff59e797ea522e8de65f38f
I20260929 18:46:12.350065 26498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43: Bootstrap starting.
I20260929 18:46:12.352398 26498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:12.353091 26498 log.cc:824] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:12.354954 26498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43: No bootstrap required, opened a new log
I20260929 18:46:12.358925 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.351178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48124 (local address 127.7.157.126:35653)
0929 18:46:12.351821 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:12.351835 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:12.351869 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:46:12.352550 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:12.352575 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:12.352972 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:12.353287 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:12.353688 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.354752 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.355939 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.356396 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.356508 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:12.357305 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:12.357310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:12.357322 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:12.357683 (+ 361us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:12.357734 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:12.357753 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:12.357826 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:12.357978 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 18:46:12.358216 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
I20260929 18:46:12.358841 26498 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a9a5de734847b9a7a9c5d128c8ce43" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35653 } }
I20260929 18:46:12.359055 26498 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:12.359125 26498 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a9a5de734847b9a7a9c5d128c8ce43, State: Initialized, Role: FOLLOWER
I20260929 18:46:12.359530 26498 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a9a5de734847b9a7a9c5d128c8ce43" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35653 } }
I20260929 18:46:12.359640 26498 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:12.359678 26498 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:12.359738 26498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:12.360544 26498 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a9a5de734847b9a7a9c5d128c8ce43" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35653 } }
I20260929 18:46:12.360869 26498 leader_election.cc:302] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5a9a5de734847b9a7a9c5d128c8ce43; no voters:
I20260929 18:46:12.361160 26498 leader_election.cc:288] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:12.361243 26503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:12.361466 26503 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [term 1 LEADER]: Becoming Leader. State: Replica: b5a9a5de734847b9a7a9c5d128c8ce43, State: Running, Role: LEADER
I20260929 18:46:12.361739 26503 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a9a5de734847b9a7a9c5d128c8ce43" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35653 } }
I20260929 18:46:12.362246 26498 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:12.363372 26505 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5a9a5de734847b9a7a9c5d128c8ce43. Latest consensus state: current_term: 1 leader_uuid: "b5a9a5de734847b9a7a9c5d128c8ce43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a9a5de734847b9a7a9c5d128c8ce43" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35653 } } }
I20260929 18:46:12.363509 26505 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:12.363711 26504 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5a9a5de734847b9a7a9c5d128c8ce43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a9a5de734847b9a7a9c5d128c8ce43" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 35653 } } }
I20260929 18:46:12.363884 26504 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43 [sys.catalog]: This master's current role is: LEADER
W20260929 18:46:12.366918 26518 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:46:12.366981 26518 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:46:12.367111 26512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:12.367766 26512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:12.373005 26512 catalog_manager.cc:1347] Generated new cluster ID: 09196f1995e7407fa19edadb37021a4f
I20260929 18:46:12.373081 26512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:12.394716 26512 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:12.396059 26512 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:12.410359 26512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43: Generated new TSK 0
I20260929 18:46:12.411398 26512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:12.416365 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35653
--builtin_ntp_servers=127.7.157.84:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:46:12.540174 26522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:12.540413 26522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:12.540449 26522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:12.544562 26522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:12.544616 26522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:12.544668 26522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:46:12.549561 26522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:12.550529 26522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:12.551875 26522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:12.560855 26527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:12.560873 26528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:12.560912 26530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:12.561118 26522 server_base.cc:1034] running on GCE node
I20260929 18:46:12.561491 26522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:12.562094 26522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:12.563256 26522 hybrid_clock.cc:648] HybridClock initialized: now 1790707572563240 us; error 29 us; skew 500 ppm
I20260929 18:46:12.564994 26522 webserver.cc:460] Webserver started at http://127.7.157.65:40279/ using document root <none> and password file <none>
I20260929 18:46:12.565505 26522 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:12.565567 26522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:12.565757 26522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:12.567909 26522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "1dd8465a51334e638feca95f461a682b"
format_stamp: "Formatted at 2026-09-29 18:46:12 on dist-test-slave-smpn"
server_key: "7410fe82d25e2f8f608e28de5763e576"
server_key_iv: "37c63339617541493ec758b68c6504c7"
server_key_version: "encryptionkey@0"
I20260929 18:46:12.568418 26522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "1dd8465a51334e638feca95f461a682b"
format_stamp: "Formatted at 2026-09-29 18:46:12 on dist-test-slave-smpn"
server_key: "7410fe82d25e2f8f608e28de5763e576"
server_key_iv: "37c63339617541493ec758b68c6504c7"
server_key_version: "encryptionkey@0"
I20260929 18:46:12.571856 26522 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:46:12.574119 26536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:12.575299 26522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:12.575402 26522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "1dd8465a51334e638feca95f461a682b"
format_stamp: "Formatted at 2026-09-29 18:46:12 on dist-test-slave-smpn"
server_key: "7410fe82d25e2f8f608e28de5763e576"
server_key_iv: "37c63339617541493ec758b68c6504c7"
server_key_version: "encryptionkey@0"
I20260929 18:46:12.575496 26522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:12.591660 26522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:12.592311 26522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:12.592460 26522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:12.593070 26522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:12.593958 26522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:12.594002 26522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:12.594049 26522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:12.594069 26522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:12.601841 26522 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:41079
I20260929 18:46:12.601897 26649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:41079 every 8 connection(s)
I20260929 18:46:12.602697 26522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:46:12.602978 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 26522
I20260929 18:46:12.603121 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:46:12.603413 7797 external_mini_cluster.cc:1428] Setting key 5e3ad4a8f87405a54aa402f47d49cf5c
I20260929 18:46:12.605571 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35653
--builtin_ntp_servers=127.7.157.84:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:46:12.612813 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.606178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:41349 (local address 127.7.157.126:35653)
0929 18:46:12.606324 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:12.606329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:12.606344 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:46:12.607720 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:12.607724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:12.607772 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:12.607841 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:12.608818 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.609640 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.610765 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.611143 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.611258 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:12.612273 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:12.612277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:12.612281 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:12.612297 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:12.612321 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:12.612332 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:12.612411 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:12.612579 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 18:46:12.612648 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:46:12.613641 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.605128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:35653 (local address 127.7.157.65:41349)
0929 18:46:12.605894 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:12.605936 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:12.607462 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:12.608046 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:12.608054 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:12.608154 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:12.608677 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:12.608690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.609757 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.609759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:12.610644 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:12.610654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.611287 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.611290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:12.611435 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:12.612011 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:46:12.612013 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:12.612054 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:46:12.612119 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:12.612409 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:12.612414 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:12.612417 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:46:12.612558 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 18:46:12.612790 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":137,"threads_started":1}
I20260929 18:46:12.614853 26650 heartbeater.cc:347] Connected to a master server at 127.7.157.126:35653
I20260929 18:46:12.615118 26650 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:12.615630 26650 heartbeater.cc:511] Master 127.7.157.126:35653 requested a full tablet report, sending...
I20260929 18:46:12.617178 26462 ts_manager.cc:194] Registered new tserver with Master: 1dd8465a51334e638feca95f461a682b (127.7.157.65:41079)
I20260929 18:46:12.618724 26462 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.157.65:41349
W20260929 18:46:12.715416 26653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:12.715679 26653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:12.715715 26653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:12.719622 26653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:12.719674 26653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:12.719725 26653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:46:12.724419 26653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:12.725446 26653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:12.726819 26653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:12.736204 26658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:12.736326 26653 server_base.cc:1034] running on GCE node
W20260929 18:46:12.736191 26659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:12.736191 26661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:12.736837 26653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:12.737478 26653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:12.738642 26653 hybrid_clock.cc:648] HybridClock initialized: now 1790707572738627 us; error 27 us; skew 500 ppm
I20260929 18:46:12.740501 26653 webserver.cc:460] Webserver started at http://127.7.157.66:44685/ using document root <none> and password file <none>
I20260929 18:46:12.741071 26653 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:12.741140 26653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:12.741333 26653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:12.743463 26653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "e47b5dae943e4fb7b28cc530806b2097"
format_stamp: "Formatted at 2026-09-29 18:46:12 on dist-test-slave-smpn"
server_key: "18226096de2f6b4a46d84615207be2e0"
server_key_iv: "dea8445e33b8aead1711a4caffe2f7ac"
server_key_version: "encryptionkey@0"
I20260929 18:46:12.744012 26653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "e47b5dae943e4fb7b28cc530806b2097"
format_stamp: "Formatted at 2026-09-29 18:46:12 on dist-test-slave-smpn"
server_key: "18226096de2f6b4a46d84615207be2e0"
server_key_iv: "dea8445e33b8aead1711a4caffe2f7ac"
server_key_version: "encryptionkey@0"
I20260929 18:46:12.747615 26653 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:46:12.749933 26667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:12.751089 26653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:12.751207 26653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "e47b5dae943e4fb7b28cc530806b2097"
format_stamp: "Formatted at 2026-09-29 18:46:12 on dist-test-slave-smpn"
server_key: "18226096de2f6b4a46d84615207be2e0"
server_key_iv: "dea8445e33b8aead1711a4caffe2f7ac"
server_key_version: "encryptionkey@0"
I20260929 18:46:12.751312 26653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:12.768342 26653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:12.769081 26653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:12.769260 26653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:12.769858 26653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:12.770803 26653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:12.770850 26653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:12.770902 26653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:12.770928 26653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:12.779163 26653 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:38039
I20260929 18:46:12.779217 26780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:38039 every 8 connection(s)
I20260929 18:46:12.780118 26653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:46:12.782174 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 26653
I20260929 18:46:12.782272 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:46:12.782531 7797 external_mini_cluster.cc:1428] Setting key 32084abcf40541606cf26c3f0a51c8ca
I20260929 18:46:12.784678 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35653
--builtin_ntp_servers=127.7.157.84:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:46:12.790058 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.781871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:53007 (local address 127.7.157.126:35653)
0929 18:46:12.782023 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:12.782028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:12.783249 (+ 1221us) server_negotiation.cc:407] Connection header received
0929 18:46:12.784844 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:12.784848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:12.784897 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:12.784973 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:12.786117 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.786885 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.788053 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.788289 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.788372 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:12.789578 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:12.789583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:12.789586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:12.789614 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:12.789637 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:12.789648 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:12.789708 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:12.789879 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 18:46:12.789932 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:46:12.791287 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.782316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:35653 (local address 127.7.157.66:53007)
0929 18:46:12.783086 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:12.783140 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:12.784607 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:12.785175 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:12.785184 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:12.785293 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:12.785957 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:12.785978 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.787015 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.787017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:12.787949 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:12.787959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.788447 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.788450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:12.788645 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:12.789227 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:46:12.789229 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:12.789285 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:46:12.789370 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:12.789733 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:12.789743 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:12.789749 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:46:12.789888 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 18:46:12.790210 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":118,"threads_started":1}
I20260929 18:46:12.792413 26781 heartbeater.cc:347] Connected to a master server at 127.7.157.126:35653
I20260929 18:46:12.792650 26781 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:12.793164 26781 heartbeater.cc:511] Master 127.7.157.126:35653 requested a full tablet report, sending...
I20260929 18:46:12.794189 26462 ts_manager.cc:194] Registered new tserver with Master: e47b5dae943e4fb7b28cc530806b2097 (127.7.157.66:38039)
I20260929 18:46:12.794770 26462 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.157.66:53007
W20260929 18:46:12.895294 26784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:12.895550 26784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:12.895591 26784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:12.899672 26784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:12.899724 26784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:12.899791 26784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:46:12.904538 26784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:35653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:12.905649 26784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:12.907050 26784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:12.917451 26792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:12.917434 26789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:12.917434 26790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:12.917773 26784 server_base.cc:1034] running on GCE node
I20260929 18:46:12.918175 26784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:12.918807 26784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:12.919967 26784 hybrid_clock.cc:648] HybridClock initialized: now 1790707572919920 us; error 51 us; skew 500 ppm
I20260929 18:46:12.921720 26784 webserver.cc:460] Webserver started at http://127.7.157.67:38129/ using document root <none> and password file <none>
I20260929 18:46:12.922277 26784 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:12.922343 26784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:12.922528 26784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:12.924548 26784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "2a933c4838e344eda0234abaebac9b9d"
format_stamp: "Formatted at 2026-09-29 18:46:12 on dist-test-slave-smpn"
server_key: "cf8ddc8f05e67306e659ab484613e597"
server_key_iv: "6d5848eb8a130bff85103967ea0771e0"
server_key_version: "encryptionkey@0"
I20260929 18:46:12.925100 26784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "2a933c4838e344eda0234abaebac9b9d"
format_stamp: "Formatted at 2026-09-29 18:46:12 on dist-test-slave-smpn"
server_key: "cf8ddc8f05e67306e659ab484613e597"
server_key_iv: "6d5848eb8a130bff85103967ea0771e0"
server_key_version: "encryptionkey@0"
I20260929 18:46:12.928762 26784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:46:12.931150 26798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:12.932379 26784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:46:12.932488 26784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "2a933c4838e344eda0234abaebac9b9d"
format_stamp: "Formatted at 2026-09-29 18:46:12 on dist-test-slave-smpn"
server_key: "cf8ddc8f05e67306e659ab484613e597"
server_key_iv: "6d5848eb8a130bff85103967ea0771e0"
server_key_version: "encryptionkey@0"
I20260929 18:46:12.932588 26784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:12.946018 26784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:12.946648 26784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:12.946820 26784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:12.947405 26784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:12.948295 26784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:12.948340 26784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:12.948390 26784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:12.948419 26784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:12.956589 26784 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:34771
I20260929 18:46:12.956624 26911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:34771 every 8 connection(s)
I20260929 18:46:12.957511 26784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:46:12.961930 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 26784
I20260929 18:46:12.962046 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:46:12.962337 7797 external_mini_cluster.cc:1428] Setting key e5a7f6a52fcc592ccc7381626c39cfbd
I20260929 18:46:12.967132 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.959377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:51591 (local address 127.7.157.126:35653)
0929 18:46:12.959510 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:12.959514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:12.960606 (+ 1092us) server_negotiation.cc:407] Connection header received
0929 18:46:12.962152 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:12.962157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:12.962215 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:12.962320 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:12.963502 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.964185 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.965382 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.965635 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.965715 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:12.966705 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:12.966708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:12.966710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:12.966729 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:12.966748 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:12.966756 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:12.966810 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:12.966927 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 18:46:12.966985 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:46:12.968027 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.959792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:35653 (local address 127.7.157.67:51591)
0929 18:46:12.960436 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:12.960479 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:12.961854 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:12.962550 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:12.962559 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:12.962676 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:12.963366 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:12.963396 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.964302 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.964306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:12.965276 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:12.965286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.965757 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.965759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:12.965900 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:12.966452 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:46:12.966454 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:12.966499 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:46:12.966572 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:12.966819 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:12.966825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:12.966829 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:46:12.966923 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 18:46:12.967217 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
I20260929 18:46:12.969069 26912 heartbeater.cc:347] Connected to a master server at 127.7.157.126:35653
I20260929 18:46:12.969281 26912 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:12.969760 26912 heartbeater.cc:511] Master 127.7.157.126:35653 requested a full tablet report, sending...
I20260929 18:46:12.970711 26462 ts_manager.cc:194] Registered new tserver with Master: 2a933c4838e344eda0234abaebac9b9d (127.7.157.67:34771)
I20260929 18:46:12.971287 26462 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.157.67:51591
I20260929 18:46:12.976053 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:46:12.981191 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.977868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48134 (local address 127.7.157.126:35653)
0929 18:46:12.977977 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:12.977981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:12.978136 (+ 155us) server_negotiation.cc:407] Connection header received
0929 18:46:12.978221 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:12.978223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:12.978260 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:12.978323 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:12.978558 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.979053 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.979878 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.980092 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.980161 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:12.980854 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:12.980856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:12.980858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:12.980868 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:12.980882 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:12.980889 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:12.980965 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:12.981055 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 18:46:12.981088 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:46:12.983731 26462 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:46:12.986536 26462 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:46:13.000371 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.994714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:38039 (local address 127.0.0.1:56734)
0929 18:46:12.995121 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:12.995136 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:12.995244 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:12.996652 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:12.996655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:12.996671 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:12.996766 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:12.996771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.997996 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.997999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:12.998781 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:12.998790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.999809 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.999813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:12.999929 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:12.999974 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 18:46:13.000091 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 18:46:13.000218 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":97,"threads_started":1}
I20260929 18:46:13.000581 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.995248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:34771 (local address 127.0.0.1:49776)
0929 18:46:12.995723 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:12.995735 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:12.995815 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:12.996498 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:12.996501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:12.996518 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:12.996602 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:12.996608 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.997629 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.997633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:12.999116 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:12.999126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:13.000140 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:13.000143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:13.000231 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:13.000255 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:46:13.000309 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 18:46:13.000385 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":98,"thread_start_us":97,"threads_started":1}
I20260929 18:46:13.000782 26928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.995560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.7.157.67:34771)
0929 18:46:12.996044 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:12.996056 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:12.996079 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:46:12.996153 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:12.996158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:12.996360 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:12.996506 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:12.996718 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.997514 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.999246 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:13.000013 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:13.000108 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:13.000279 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:13.000336 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 18:46:13.000429 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":116,"threads_started":1}
I20260929 18:46:13.001000 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.994715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34832 (local address 127.7.157.65:41079)
0929 18:46:12.995516 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:12.995528 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:12.995555 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:46:12.995630 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:12.995635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:12.995854 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:12.996352 (+ 498us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:12.996492 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.997623 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.999339 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:13.000095 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:13.000463 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:13.000670 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:13.000724 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 18:46:13.000823 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":102,"threads_started":1}
I20260929 18:46:13.001071 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.994581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:41079 (local address 127.0.0.1:34832)
0929 18:46:12.995055 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:12.995125 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:12.995244 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:12.996036 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:12.996043 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:12.996067 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:12.996205 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:12.996212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.997727 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.997730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:12.999214 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:12.999223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:13.000209 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:13.000212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:13.000286 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:13.000306 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:46:13.000343 (+ 37us) client_negotiation.cc:240] Negotiation successful
0929 18:46:13.000394 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":123,"thread_start_us":85,"threads_started":1}
I20260929 18:46:13.001514 26927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:12.995393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.7.157.66:38039)
0929 18:46:12.996112 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:12.996122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:12.996143 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:46:12.996211 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:12.996216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:12.996410 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:12.996565 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:12.996871 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.997890 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.998901 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:12.999706 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:12.999809 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:13.000006 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:13.001217 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0929 18:46:13.001337 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":109,"threads_started":1}
I20260929 18:46:13.003808 26715 tablet_service.cc:1467] Processing CreateTablet for tablet a30b832d9dc5483da84ff670cdedad6c (DEFAULT_TABLE table=test-table [id=1588f3ef34734bc4bb25fd3481dc1f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:13.003824 26584 tablet_service.cc:1467] Processing CreateTablet for tablet a30b832d9dc5483da84ff670cdedad6c (DEFAULT_TABLE table=test-table [id=1588f3ef34734bc4bb25fd3481dc1f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:13.003823 26846 tablet_service.cc:1467] Processing CreateTablet for tablet a30b832d9dc5483da84ff670cdedad6c (DEFAULT_TABLE table=test-table [id=1588f3ef34734bc4bb25fd3481dc1f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:13.004796 26715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a30b832d9dc5483da84ff670cdedad6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:13.004808 26584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a30b832d9dc5483da84ff670cdedad6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:13.004808 26846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a30b832d9dc5483da84ff670cdedad6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:13.010990 26929 tablet_bootstrap.cc:492] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097: Bootstrap starting.
I20260929 18:46:13.012882 26930 tablet_bootstrap.cc:492] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d: Bootstrap starting.
I20260929 18:46:13.012882 26931 tablet_bootstrap.cc:492] T a30b832d9dc5483da84ff670cdedad6c P 1dd8465a51334e638feca95f461a682b: Bootstrap starting.
I20260929 18:46:13.014479 26929 tablet_bootstrap.cc:654] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:13.015061 26931 tablet_bootstrap.cc:654] T a30b832d9dc5483da84ff670cdedad6c P 1dd8465a51334e638feca95f461a682b: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:13.015061 26930 tablet_bootstrap.cc:654] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:13.015270 26929 log.cc:824] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:13.015921 26931 log.cc:824] T a30b832d9dc5483da84ff670cdedad6c P 1dd8465a51334e638feca95f461a682b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:13.015920 26930 log.cc:824] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:13.017220 26929 tablet_bootstrap.cc:492] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097: No bootstrap required, opened a new log
I20260929 18:46:13.017467 26929 ts_tablet_manager.cc:1397] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260929 18:46:13.017835 26931 tablet_bootstrap.cc:492] T a30b832d9dc5483da84ff670cdedad6c P 1dd8465a51334e638feca95f461a682b: No bootstrap required, opened a new log
I20260929 18:46:13.017831 26930 tablet_bootstrap.cc:492] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d: No bootstrap required, opened a new log
I20260929 18:46:13.018029 26930 ts_tablet_manager.cc:1397] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:46:13.018057 26931 ts_tablet_manager.cc:1397] T a30b832d9dc5483da84ff670cdedad6c P 1dd8465a51334e638feca95f461a682b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:46:13.020828 26929 raft_consensus.cc:348] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd8465a51334e638feca95f461a682b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41079 } } peers { permanent_uuid: "e47b5dae943e4fb7b28cc530806b2097" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38039 } } peers { permanent_uuid: "2a933c4838e344eda0234abaebac9b9d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 34771 } }
I20260929 18:46:13.021066 26929 raft_consensus.cc:374] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:13.021143 26929 raft_consensus.cc:729] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47b5dae943e4fb7b28cc530806b2097, State: Initialized, Role: FOLLOWER
I20260929 18:46:13.021569 26929 consensus_queue.cc:260] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd8465a51334e638feca95f461a682b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41079 } } peers { permanent_uuid: "e47b5dae943e4fb7b28cc530806b2097" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38039 } } peers { permanent_uuid: "2a933c4838e344eda0234abaebac9b9d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 34771 } }
I20260929 18:46:13.022182 26781 heartbeater.cc:503] Master 127.7.157.126:35653 was elected leader, sending a full tablet report...
I20260929 18:46:13.022167 26931 raft_consensus.cc:348] T a30b832d9dc5483da84ff670cdedad6c P 1dd8465a51334e638feca95f461a682b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd8465a51334e638feca95f461a682b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41079 } } peers { permanent_uuid: "e47b5dae943e4fb7b28cc530806b2097" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38039 } } peers { permanent_uuid: "2a933c4838e344eda0234abaebac9b9d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 34771 } }
I20260929 18:46:13.022167 26930 raft_consensus.cc:348] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd8465a51334e638feca95f461a682b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41079 } } peers { permanent_uuid: "e47b5dae943e4fb7b28cc530806b2097" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38039 } } peers { permanent_uuid: "2a933c4838e344eda0234abaebac9b9d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 34771 } }
I20260929 18:46:13.022430 26930 raft_consensus.cc:374] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:13.022430 26931 raft_consensus.cc:374] T a30b832d9dc5483da84ff670cdedad6c P 1dd8465a51334e638feca95f461a682b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:13.022488 26931 raft_consensus.cc:729] T a30b832d9dc5483da84ff670cdedad6c P 1dd8465a51334e638feca95f461a682b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd8465a51334e638feca95f461a682b, State: Initialized, Role: FOLLOWER
I20260929 18:46:13.022529 26930 raft_consensus.cc:729] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a933c4838e344eda0234abaebac9b9d, State: Initialized, Role: FOLLOWER
I20260929 18:46:13.022956 26930 consensus_queue.cc:260] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd8465a51334e638feca95f461a682b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41079 } } peers { permanent_uuid: "e47b5dae943e4fb7b28cc530806b2097" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38039 } } peers { permanent_uuid: "2a933c4838e344eda0234abaebac9b9d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 34771 } }
I20260929 18:46:13.022956 26931 consensus_queue.cc:260] T a30b832d9dc5483da84ff670cdedad6c P 1dd8465a51334e638feca95f461a682b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd8465a51334e638feca95f461a682b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41079 } } peers { permanent_uuid: "e47b5dae943e4fb7b28cc530806b2097" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38039 } } peers { permanent_uuid: "2a933c4838e344eda0234abaebac9b9d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 34771 } }
I20260929 18:46:13.023752 26929 ts_tablet_manager.cc:1428] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260929 18:46:13.023947 26931 ts_tablet_manager.cc:1428] T a30b832d9dc5483da84ff670cdedad6c P 1dd8465a51334e638feca95f461a682b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:46:13.024358 26650 heartbeater.cc:503] Master 127.7.157.126:35653 was elected leader, sending a full tablet report...
I20260929 18:46:13.024593 26930 ts_tablet_manager.cc:1428] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:46:13.024904 26912 heartbeater.cc:503] Master 127.7.157.126:35653 was elected leader, sending a full tablet report...
W20260929 18:46:13.031001 26782 tablet.cc:2369] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:46:13.104092 26651 tablet.cc:2369] T a30b832d9dc5483da84ff670cdedad6c P 1dd8465a51334e638feca95f461a682b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:46:13.161752 26937 raft_consensus.cc:482] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:46:13.161991 26937 raft_consensus.cc:504] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd8465a51334e638feca95f461a682b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41079 } } peers { permanent_uuid: "e47b5dae943e4fb7b28cc530806b2097" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38039 } } peers { permanent_uuid: "2a933c4838e344eda0234abaebac9b9d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 34771 } }
I20260929 18:46:13.163269 26937 leader_election.cc:288] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dd8465a51334e638feca95f461a682b (127.7.157.65:41079), e47b5dae943e4fb7b28cc530806b2097 (127.7.157.66:38039)
I20260929 18:46:13.167011 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:13.163536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:41079 (local address 127.7.157.67:48687)
0929 18:46:13.163668 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:13.163684 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:13.163779 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:13.164064 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:13.164067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:13.164088 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:13.164230 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:13.164237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:13.165060 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:13.165063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:13.165946 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:13.165956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:13.166623 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:13.166627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:13.166724 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:13.166749 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:46:13.166814 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 18:46:13.166885 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"spinlock_wait_cycles":4480}
I20260929 18:46:13.167114 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:13.163680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:48687 (local address 127.7.157.65:41079)
0929 18:46:13.163809 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:13.163813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:13.163829 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:46:13.163889 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:13.163892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:13.163942 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:13.164037 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:13.164342 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:13.164948 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:13.166068 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:13.166511 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:13.166610 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:13.166650 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:13.166842 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 18:46:13.166927 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:46:13.167200 26938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:13.163547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:38039 (local address 127.7.157.67:39523)
0929 18:46:13.164011 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:13.164025 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:13.164140 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:13.164377 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:13.164392 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:13.164410 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:13.164542 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:13.164549 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:13.165284 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:13.165286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:13.166104 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:13.166115 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:13.166780 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:13.166783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:13.166884 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:13.166905 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:46:13.166978 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:46:13.167043 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":126,"spinlock_wait_cycles":27648,"thread_start_us":123,"threads_started":1}
I20260929 18:46:13.167240 26927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:13.163680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:39523 (local address 127.7.157.66:38039)
0929 18:46:13.163809 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:13.163814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:13.164108 (+ 294us) server_negotiation.cc:407] Connection header received
0929 18:46:13.164233 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:13.164236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:13.164289 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:13.164360 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:13.164648 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:13.165176 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:13.166224 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:13.166650 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:13.166763 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:13.166808 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:13.167006 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 18:46:13.167127 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:46:13.167906 26604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a30b832d9dc5483da84ff670cdedad6c" candidate_uuid: "2a933c4838e344eda0234abaebac9b9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd8465a51334e638feca95f461a682b" is_pre_election: true
I20260929 18:46:13.167906 26735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a30b832d9dc5483da84ff670cdedad6c" candidate_uuid: "2a933c4838e344eda0234abaebac9b9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47b5dae943e4fb7b28cc530806b2097" is_pre_election: true
I20260929 18:46:13.168195 26735 raft_consensus.cc:2454] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a933c4838e344eda0234abaebac9b9d in term 0.
I20260929 18:46:13.168201 26604 raft_consensus.cc:2454] T a30b832d9dc5483da84ff670cdedad6c P 1dd8465a51334e638feca95f461a682b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a933c4838e344eda0234abaebac9b9d in term 0.
I20260929 18:46:13.168659 26801 leader_election.cc:302] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd8465a51334e638feca95f461a682b, 2a933c4838e344eda0234abaebac9b9d; no voters:
I20260929 18:46:13.168924 26937 raft_consensus.cc:2781] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:46:13.169013 26937 raft_consensus.cc:482] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:46:13.169045 26937 raft_consensus.cc:3037] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:13.170076 26937 raft_consensus.cc:504] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd8465a51334e638feca95f461a682b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41079 } } peers { permanent_uuid: "e47b5dae943e4fb7b28cc530806b2097" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38039 } } peers { permanent_uuid: "2a933c4838e344eda0234abaebac9b9d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 34771 } }
I20260929 18:46:13.170421 26937 leader_election.cc:288] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [CANDIDATE]: Term 1 election: Requested vote from peers 1dd8465a51334e638feca95f461a682b (127.7.157.65:41079), e47b5dae943e4fb7b28cc530806b2097 (127.7.157.66:38039)
I20260929 18:46:13.170781 26735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a30b832d9dc5483da84ff670cdedad6c" candidate_uuid: "2a933c4838e344eda0234abaebac9b9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47b5dae943e4fb7b28cc530806b2097"
I20260929 18:46:13.170781 26604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a30b832d9dc5483da84ff670cdedad6c" candidate_uuid: "2a933c4838e344eda0234abaebac9b9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd8465a51334e638feca95f461a682b"
I20260929 18:46:13.170913 26604 raft_consensus.cc:3037] T a30b832d9dc5483da84ff670cdedad6c P 1dd8465a51334e638feca95f461a682b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:13.170913 26735 raft_consensus.cc:3037] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:13.172190 26604 raft_consensus.cc:2454] T a30b832d9dc5483da84ff670cdedad6c P 1dd8465a51334e638feca95f461a682b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a933c4838e344eda0234abaebac9b9d in term 1.
I20260929 18:46:13.172190 26735 raft_consensus.cc:2454] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a933c4838e344eda0234abaebac9b9d in term 1.
I20260929 18:46:13.172509 26801 leader_election.cc:302] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd8465a51334e638feca95f461a682b, 2a933c4838e344eda0234abaebac9b9d; no voters:
I20260929 18:46:13.172669 26937 raft_consensus.cc:2781] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:13.172894 26937 raft_consensus.cc:686] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [term 1 LEADER]: Becoming Leader. State: Replica: 2a933c4838e344eda0234abaebac9b9d, State: Running, Role: LEADER
I20260929 18:46:13.173223 26937 consensus_queue.cc:237] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd8465a51334e638feca95f461a682b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41079 } } peers { permanent_uuid: "e47b5dae943e4fb7b28cc530806b2097" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38039 } } peers { permanent_uuid: "2a933c4838e344eda0234abaebac9b9d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 34771 } }
I20260929 18:46:13.176371 26461 catalog_manager.cc:5495] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a933c4838e344eda0234abaebac9b9d (127.7.157.67). New cstate: current_term: 1 leader_uuid: "2a933c4838e344eda0234abaebac9b9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd8465a51334e638feca95f461a682b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e47b5dae943e4fb7b28cc530806b2097" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a933c4838e344eda0234abaebac9b9d" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 34771 } health_report { overall_health: HEALTHY } } }
I20260929 18:46:13.197465 26928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:13.194335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49786 (local address 127.7.157.67:34771)
0929 18:46:13.194450 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:13.194454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:13.194478 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:46:13.194551 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:13.194553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:13.194610 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:13.194701 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:46:13.194884 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:13.195424 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:13.196245 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:13.196440 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:13.196512 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:13.196751 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:46:13.197145 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:46:13.197250 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:13.197307 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 18:46:13.197339 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:46:13.203370 26735 raft_consensus.cc:1261] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097 [term 1 FOLLOWER]: Refusing update from remote peer 2a933c4838e344eda0234abaebac9b9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:46:13.203369 26604 raft_consensus.cc:1261] T a30b832d9dc5483da84ff670cdedad6c P 1dd8465a51334e638feca95f461a682b [term 1 FOLLOWER]: Refusing update from remote peer 2a933c4838e344eda0234abaebac9b9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:46:13.204090 26939 consensus_queue.cc:1035] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dd8465a51334e638feca95f461a682b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 41079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:46:13.204264 26937 consensus_queue.cc:1035] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47b5dae943e4fb7b28cc530806b2097" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 38039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:46:13.208662 26913 tablet.cc:2369] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:46:13.213212 26943 mvcc.cc:205] Tried to move back new op lower bound from 7334738219833282560 to 7334738219720253440. Current Snapshot: MvccSnapshot[applied={T|T < 7334738219833282560}]
I20260929 18:46:13.220819 26462 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:46:13.220973 26462 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:46:13.223201 26462 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5a9a5de734847b9a7a9c5d128c8ce43: Sending DeleteTablet for 3 replicas of tablet a30b832d9dc5483da84ff670cdedad6c
I20260929 18:46:13.223994 26846 tablet_service.cc:1514] Processing DeleteTablet for tablet a30b832d9dc5483da84ff670cdedad6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:13 UTC) from {username='slave'} at 127.0.0.1:49776
I20260929 18:46:13.223966 26715 tablet_service.cc:1514] Processing DeleteTablet for tablet a30b832d9dc5483da84ff670cdedad6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:13 UTC) from {username='slave'} at 127.0.0.1:56734
I20260929 18:46:13.223989 26584 tablet_service.cc:1514] Processing DeleteTablet for tablet a30b832d9dc5483da84ff670cdedad6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:13 UTC) from {username='slave'} at 127.0.0.1:34832
I20260929 18:46:13.224457 26953 tablet_replica.cc:330] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097: stopping tablet replica
I20260929 18:46:13.224570 26952 tablet_replica.cc:330] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d: stopping tablet replica
I20260929 18:46:13.224855 26952 raft_consensus.cc:2229] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:46:13.224855 26953 raft_consensus.cc:2229] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:46:13.225164 26953 raft_consensus.cc:2258] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:46:13.225279 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 26522
I20260929 18:46:13.225282 26952 raft_consensus.cc:2258] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:46:13.227304 26952 ts_tablet_manager.cc:1905] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:46:13.227427 26953 ts_tablet_manager.cc:1905] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:46:13.229934 26953 ts_tablet_manager.cc:1918] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:46:13.229934 26952 ts_tablet_manager.cc:1918] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:46:13.230036 26952 log.cc:1196] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/a30b832d9dc5483da84ff670cdedad6c
I20260929 18:46:13.230113 26953 log.cc:1196] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/a30b832d9dc5483da84ff670cdedad6c
I20260929 18:46:13.230391 26952 ts_tablet_manager.cc:1939] T a30b832d9dc5483da84ff670cdedad6c P 2a933c4838e344eda0234abaebac9b9d: Deleting consensus metadata
I20260929 18:46:13.230439 26953 ts_tablet_manager.cc:1939] T a30b832d9dc5483da84ff670cdedad6c P e47b5dae943e4fb7b28cc530806b2097: Deleting consensus metadata
I20260929 18:46:13.231609 26446 catalog_manager.cc:4841] TS 2a933c4838e344eda0234abaebac9b9d (127.7.157.67:34771): tablet a30b832d9dc5483da84ff670cdedad6c (table test-table [id=1588f3ef34734bc4bb25fd3481dc1f08]) successfully deleted
I20260929 18:46:13.231609 26447 catalog_manager.cc:4841] TS e47b5dae943e4fb7b28cc530806b2097 (127.7.157.66:38039): tablet a30b832d9dc5483da84ff670cdedad6c (table test-table [id=1588f3ef34734bc4bb25fd3481dc1f08]) successfully deleted
W20260929 18:46:13.233495 26448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:41079 (error 108)
I20260929 18:46:13.234280 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 26653
I20260929 18:46:13.234465 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:13.234171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:41079 (local address 127.0.0.1:34834)
0929 18:46:13.234320 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:13.234386 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 18:46:13.234972 26448 catalog_manager.cc:4584] TS 1dd8465a51334e638feca95f461a682b (127.7.157.65:41079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a30b832d9dc5483da84ff670cdedad6c: Network error: Client connection negotiation failed: client connection to 127.7.157.65:41079: connect: Connection refused (error 111)
I20260929 18:46:13.239398 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 26784
I20260929 18:46:13.244064 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 26431
2026-09-29T18:46:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1106 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:13 dist-test-slave-smpn krb5kdc[26958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:13 dist-test-slave-smpn krb5kdc[26958](info): set up 2 sockets
Sep 29 18:46:13 dist-test-slave-smpn krb5kdc[26958](info): commencing operation
krb5kdc: starting...
W20260929 18:46:14.194942 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:14 dist-test-slave-smpn krb5kdc[26958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:46:14.310590 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:33349
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:33349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:14.418025 26971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:14.418294 26971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:14.418346 26971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:14.422370 26971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:14.422443 26971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:14.422469 26971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:14.422492 26971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:14.422509 26971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:14.427831 26971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:33349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:33349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:14.428938 26971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:14.429980 26971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:46:14.438544 26971 server_base.cc:1034] running on GCE node
W20260929 18:46:14.438490 26979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:14.438504 26977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:14.438490 26976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:14.439363 26971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:14.440558 26971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:14.441722 26971 hybrid_clock.cc:648] HybridClock initialized: now 1790707574441697 us; error 39 us; skew 500 ppm
Sep 29 18:46:14 dist-test-slave-smpn krb5kdc[26958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707574, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:14.444358 26971 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:46:14.445298 26971 webserver.cc:460] Webserver started at http://127.7.157.126:39021/ using document root <none> and password file <none>
I20260929 18:46:14.445839 26971 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:14.445906 26971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:14.446132 26971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:14.448266 26971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "1d49ce6db91846b1b63bd95a0c94e195"
format_stamp: "Formatted at 2026-09-29 18:46:14 on dist-test-slave-smpn"
server_key: "36a958d6e16aced418007acfe45677ce"
server_key_iv: "26a7fda586543668fff275a14a970bd7"
server_key_version: "encryptionkey@0"
I20260929 18:46:14.448825 26971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "1d49ce6db91846b1b63bd95a0c94e195"
format_stamp: "Formatted at 2026-09-29 18:46:14 on dist-test-slave-smpn"
server_key: "36a958d6e16aced418007acfe45677ce"
server_key_iv: "26a7fda586543668fff275a14a970bd7"
server_key_version: "encryptionkey@0"
I20260929 18:46:14.452600 26971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:46:14.455022 26986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:14.456260 26971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:14.456374 26971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "1d49ce6db91846b1b63bd95a0c94e195"
format_stamp: "Formatted at 2026-09-29 18:46:14 on dist-test-slave-smpn"
server_key: "36a958d6e16aced418007acfe45677ce"
server_key_iv: "26a7fda586543668fff275a14a970bd7"
server_key_version: "encryptionkey@0"
I20260929 18:46:14.456472 26971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:14.473873 26971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:14.476609 26971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:14.476783 26971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:14.483753 26971 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:33349
I20260929 18:46:14.483769 27038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:33349 every 8 connection(s)
I20260929 18:46:14.484705 26971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:14.485970 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 26971
I20260929 18:46:14.486100 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:14.486400 7797 external_mini_cluster.cc:1428] Setting key 1c8372fccb40e4fe322a50e5ce7c5de4
I20260929 18:46:14.487972 27039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:14.494814 27039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195: Bootstrap starting.
Sep 29 18:46:14 dist-test-slave-smpn krb5kdc[26958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:14.497349 27039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:14.498066 27039 log.cc:824] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:14.500327 27039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195: No bootstrap required, opened a new log
I20260929 18:46:14.500955 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:14.487787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.7.157.126:33349)
0929 18:46:14.488195 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:14.488211 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:14.488249 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:46:14.488887 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:14.488905 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:14.489269 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:14.489569 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:14.490437 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:14.491499 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:14.492633 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:14.493180 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:14.493326 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:14.496360 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:14.496365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:14.496380 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:14.496411 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:14.498037 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:14.498517 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:14.498523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:14.498530 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:14.498670 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:14.498914 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:14.498916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:14.498917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:14.499377 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:14.499528 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:14.499823 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 18:46:14.500111 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260929 18:46:14.503799 27039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d49ce6db91846b1b63bd95a0c94e195" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33349 } }
I20260929 18:46:14.504011 27039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:14.504060 27039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d49ce6db91846b1b63bd95a0c94e195, State: Initialized, Role: FOLLOWER
I20260929 18:46:14.504449 27039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d49ce6db91846b1b63bd95a0c94e195" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33349 } }
I20260929 18:46:14.504603 27039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:14.504716 27039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:14.504830 27039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:14.505659 27039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d49ce6db91846b1b63bd95a0c94e195" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33349 } }
I20260929 18:46:14.505955 27039 leader_election.cc:302] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d49ce6db91846b1b63bd95a0c94e195; no voters:
I20260929 18:46:14.506199 27039 leader_election.cc:288] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:14.506323 27044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:14.506541 27044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [term 1 LEADER]: Becoming Leader. State: Replica: 1d49ce6db91846b1b63bd95a0c94e195, State: Running, Role: LEADER
I20260929 18:46:14.506855 27044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d49ce6db91846b1b63bd95a0c94e195" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33349 } }
I20260929 18:46:14.507361 27039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:14.508267 27045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d49ce6db91846b1b63bd95a0c94e195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d49ce6db91846b1b63bd95a0c94e195" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33349 } } }
I20260929 18:46:14.508543 27046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d49ce6db91846b1b63bd95a0c94e195. Latest consensus state: current_term: 1 leader_uuid: "1d49ce6db91846b1b63bd95a0c94e195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d49ce6db91846b1b63bd95a0c94e195" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33349 } } }
I20260929 18:46:14.508627 27045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:14.508677 27046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:14.509569 27054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:14.512233 27054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:14.517619 27054 catalog_manager.cc:1347] Generated new cluster ID: 146e66b127dd49c9be5498fb93950b8e
I20260929 18:46:14.517683 27054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:14.551090 27054 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:14.552417 27054 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:14.557327 27054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195: Generated new TSK 0
I20260929 18:46:14.558130 27054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.157.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
Entry for principal kudu/127.7.157.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab.
I20260929 18:46:14.589546 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:33349
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:14.699800 27065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:14.700090 27065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:14.700150 27065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:14.704206 27065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:14.704272 27065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:14.704350 27065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:46:14.709734 27065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.65.keytab
--principal=kudu/127.7.157.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:14.710788 27065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:14.711864 27065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:14.721158 27071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:14.721171 27070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:14.721171 27073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:14.721239 27065 server_base.cc:1034] running on GCE node
I20260929 18:46:14.721747 27065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:14.722355 27065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:14.723512 27065 hybrid_clock.cc:648] HybridClock initialized: now 1790707574723495 us; error 26 us; skew 500 ppm
Sep 29 18:46:14 dist-test-slave-smpn krb5kdc[26958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707574, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:14.726085 27065 init.cc:377] Logged in from keytab as kudu/127.7.157.65@KRBTEST.COM (short username kudu)
I20260929 18:46:14.726945 27065 webserver.cc:460] Webserver started at http://127.7.157.65:41923/ using document root <none> and password file <none>
I20260929 18:46:14.727516 27065 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:14.727578 27065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:14.727782 27065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:14.729805 27065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "3845db93e2be4479b66ea2b11e4a432b"
format_stamp: "Formatted at 2026-09-29 18:46:14 on dist-test-slave-smpn"
server_key: "41e6a9dc305fbc2c3f4ee05371ad3829"
server_key_iv: "2649f08ed464f9671cad5b665a4b824a"
server_key_version: "encryptionkey@0"
I20260929 18:46:14.730337 27065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "3845db93e2be4479b66ea2b11e4a432b"
format_stamp: "Formatted at 2026-09-29 18:46:14 on dist-test-slave-smpn"
server_key: "41e6a9dc305fbc2c3f4ee05371ad3829"
server_key_iv: "2649f08ed464f9671cad5b665a4b824a"
server_key_version: "encryptionkey@0"
I20260929 18:46:14.733896 27065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:46:14.736300 27080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:14.737471 27065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:46:14.737568 27065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "3845db93e2be4479b66ea2b11e4a432b"
format_stamp: "Formatted at 2026-09-29 18:46:14 on dist-test-slave-smpn"
server_key: "41e6a9dc305fbc2c3f4ee05371ad3829"
server_key_iv: "2649f08ed464f9671cad5b665a4b824a"
server_key_version: "encryptionkey@0"
I20260929 18:46:14.737668 27065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:14.760169 27065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:14.763226 27065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:14.763422 27065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:14.764221 27065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:14.765200 27065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:14.765249 27065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:14.765295 27065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:14.765326 27065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:14.774045 27065 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:34195
I20260929 18:46:14.774075 27193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:34195 every 8 connection(s)
I20260929 18:46:14.774981 27065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:46:14.775844 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 27065
I20260929 18:46:14.775967 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:46:14.776227 7797 external_mini_cluster.cc:1428] Setting key 6bcc83f61a7596061564ca795b871203
Sep 29 18:46:14 dist-test-slave-smpn krb5kdc[26958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707574, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.65@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.157.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.66@KRBTEST.COM" created.
I20260929 18:46:14.788525 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:14.776873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:57215 (local address 127.7.157.126:33349)
0929 18:46:14.777011 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:14.777016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:14.778036 (+ 1020us) server_negotiation.cc:407] Connection header received
0929 18:46:14.778917 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:14.778921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:14.778976 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:14.779053 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:14.780608 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:14.781366 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:14.782572 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:14.782937 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:14.783051 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:14.785375 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:14.785379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:14.785382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:14.785410 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:14.786625 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:14.787094 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:14.787098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:14.787100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:14.787171 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:14.787522 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:14.787524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:14.787525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:14.787684 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:14.787753 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:14.788279 (+ 526us) server_negotiation.cc:299] Negotiation successful
0929 18:46:14.788390 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:46:14.789655 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:14.777355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:33349 (local address 127.7.157.65:57215)
0929 18:46:14.777884 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:14.777930 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:14.778722 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:14.779273 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:14.779284 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:14.779773 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:14.780453 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:14.780484 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:14.781501 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:14.781504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:14.782444 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:14.782454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:14.783101 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:14.783105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:14.783246 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:14.783815 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:14.783831 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:14.785207 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:14.786723 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:14.786728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:14.786741 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:14.786996 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:14.787238 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:14.787240 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:14.787242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:14.787450 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:14.787756 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:14.787763 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:14.788042 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 18:46:14.788235 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 18:46:14.788580 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":145,"threads_started":1}
I20260929 18:46:14.791093 27194 heartbeater.cc:347] Connected to a master server at 127.7.157.126:33349
I20260929 18:46:14.791355 27194 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:14.791968 27194 heartbeater.cc:511] Master 127.7.157.126:33349 requested a full tablet report, sending...
I20260929 18:46:14.793617 27003 ts_manager.cc:194] Registered new tserver with Master: 3845db93e2be4479b66ea2b11e4a432b (127.7.157.65:34195)
I20260929 18:46:14.794899 27003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.65@KRBTEST.COM'} at 127.7.157.65:57215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
Entry for principal kudu/127.7.157.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab.
I20260929 18:46:14.804479 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:33349
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:14.911805 27199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:14.912060 27199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:14.912108 27199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:14.916023 27199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:14.916074 27199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:14.916147 27199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:46:14.921466 27199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.66.keytab
--principal=kudu/127.7.157.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:14.922502 27199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:14.923537 27199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:14.932897 27207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:14.933048 27199 server_base.cc:1034] running on GCE node
W20260929 18:46:14.932898 27204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:14.932914 27205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:14.933560 27199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:14.934177 27199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:14.935325 27199 hybrid_clock.cc:648] HybridClock initialized: now 1790707574935302 us; error 35 us; skew 500 ppm
Sep 29 18:46:14 dist-test-slave-smpn krb5kdc[26958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707574, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:14.937916 27199 init.cc:377] Logged in from keytab as kudu/127.7.157.66@KRBTEST.COM (short username kudu)
I20260929 18:46:14.938776 27199 webserver.cc:460] Webserver started at http://127.7.157.66:32993/ using document root <none> and password file <none>
I20260929 18:46:14.939353 27199 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:14.939416 27199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:14.939592 27199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:14.941669 27199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "bd87e01636e74d1cbffdbfb749e50b8c"
format_stamp: "Formatted at 2026-09-29 18:46:14 on dist-test-slave-smpn"
server_key: "f87ef1732da27eb3d6f1122a02e0f080"
server_key_iv: "0cd2a5ae5e8624444c4438ff6167c9e7"
server_key_version: "encryptionkey@0"
I20260929 18:46:14.942179 27199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "bd87e01636e74d1cbffdbfb749e50b8c"
format_stamp: "Formatted at 2026-09-29 18:46:14 on dist-test-slave-smpn"
server_key: "f87ef1732da27eb3d6f1122a02e0f080"
server_key_iv: "0cd2a5ae5e8624444c4438ff6167c9e7"
server_key_version: "encryptionkey@0"
I20260929 18:46:14.945849 27199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 18:46:14.948305 27214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:14.949528 27199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:46:14.949628 27199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "bd87e01636e74d1cbffdbfb749e50b8c"
format_stamp: "Formatted at 2026-09-29 18:46:14 on dist-test-slave-smpn"
server_key: "f87ef1732da27eb3d6f1122a02e0f080"
server_key_iv: "0cd2a5ae5e8624444c4438ff6167c9e7"
server_key_version: "encryptionkey@0"
I20260929 18:46:14.949715 27199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:14.969098 27199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:14.972218 27199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:14.972398 27199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:14.972982 27199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:14.973829 27199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:14.973876 27199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:14.973918 27199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:14.973946 27199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:14.981848 27199 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:35129
I20260929 18:46:14.981885 27327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:35129 every 8 connection(s)
I20260929 18:46:14.982723 27199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:46:14.990197 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 27199
I20260929 18:46:14.990340 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:46:14.990603 7797 external_mini_cluster.cc:1428] Setting key d254db5907885499fcdb380028cadaaa
Sep 29 18:46:14 dist-test-slave-smpn krb5kdc[26958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707574, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.66@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:14.995118 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:14.984466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:38557 (local address 127.7.157.126:33349)
0929 18:46:14.984597 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:14.984600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:14.985464 (+ 864us) server_negotiation.cc:407] Connection header received
0929 18:46:14.986305 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:14.986308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:14.986353 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:14.986424 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:14.987597 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:14.988135 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:14.989215 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:14.989418 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:14.989490 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:14.991863 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:14.991867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:14.991870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:14.991894 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:14.993181 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:14.993655 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:14.993658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:14.993660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:14.993732 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:14.994027 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:14.994030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:14.994032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:14.994242 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:14.994330 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:14.994807 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 18:46:14.994926 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:46:14.995958 27330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:14.984827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:33349 (local address 127.7.157.66:38557)
0929 18:46:14.985294 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:14.985338 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:14.986108 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:14.986583 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:14.986589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:14.986934 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:14.987480 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:14.987494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:14.988248 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:14.988251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:14.989122 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:14.989131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:14.989540 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:14.989543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:14.989683 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:14.990224 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:14.990240 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:14.991697 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:14.993294 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:14.993300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:14.993315 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:14.993563 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:14.993810 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:14.993813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:14.993815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:14.993945 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:14.994339 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:14.994345 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:14.994588 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 18:46:14.994747 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 18:46:14.995059 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":128,"threads_started":1}
I20260929 18:46:14.997035 27328 heartbeater.cc:347] Connected to a master server at 127.7.157.126:33349
I20260929 18:46:14.997254 27328 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:14.997699 27328 heartbeater.cc:511] Master 127.7.157.126:33349 requested a full tablet report, sending...
I20260929 18:46:14.998696 27003 ts_manager.cc:194] Registered new tserver with Master: bd87e01636e74d1cbffdbfb749e50b8c (127.7.157.66:35129)
WARNING: no policy specified for kudu/127.7.157.67@KRBTEST.COM; defaulting to no policy
I20260929 18:46:14.999413 27003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.66@KRBTEST.COM'} at 127.7.157.66:38557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
Entry for principal kudu/127.7.157.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab.
I20260929 18:46:15.015255 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:33349
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:15.124642 27333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:15.124943 27333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:15.124998 27333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:15.129158 27333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:15.129230 27333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:15.129309 27333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:46:15.134707 27333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.67.keytab
--principal=kudu/127.7.157.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:15.135828 27333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:15.136847 27333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:15.146595 27338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:15.146606 27333 server_base.cc:1034] running on GCE node
W20260929 18:46:15.146566 27341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:15.146584 27339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:15.147264 27333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:15.147903 27333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:15.149061 27333 hybrid_clock.cc:648] HybridClock initialized: now 1790707575149033 us; error 41 us; skew 500 ppm
Sep 29 18:46:15 dist-test-slave-smpn krb5kdc[26958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707575, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:15.151719 27333 init.cc:377] Logged in from keytab as kudu/127.7.157.67@KRBTEST.COM (short username kudu)
I20260929 18:46:15.152621 27333 webserver.cc:460] Webserver started at http://127.7.157.67:44951/ using document root <none> and password file <none>
I20260929 18:46:15.153160 27333 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:15.153224 27333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:15.153445 27333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:15.155517 27333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "d4aca2ab7672433984d09a29bead7f0a"
format_stamp: "Formatted at 2026-09-29 18:46:15 on dist-test-slave-smpn"
server_key: "183b244d06cc94d0c3dbea9036e6c1e9"
server_key_iv: "041bf57213d55d836aa16385ad5b2b0f"
server_key_version: "encryptionkey@0"
I20260929 18:46:15.156047 27333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "d4aca2ab7672433984d09a29bead7f0a"
format_stamp: "Formatted at 2026-09-29 18:46:15 on dist-test-slave-smpn"
server_key: "183b244d06cc94d0c3dbea9036e6c1e9"
server_key_iv: "041bf57213d55d836aa16385ad5b2b0f"
server_key_version: "encryptionkey@0"
I20260929 18:46:15.159559 27333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:46:15.161947 27348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:15.163127 27333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:15.163228 27333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "d4aca2ab7672433984d09a29bead7f0a"
format_stamp: "Formatted at 2026-09-29 18:46:15 on dist-test-slave-smpn"
server_key: "183b244d06cc94d0c3dbea9036e6c1e9"
server_key_iv: "041bf57213d55d836aa16385ad5b2b0f"
server_key_version: "encryptionkey@0"
I20260929 18:46:15.163316 27333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:15.188081 27333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:15.190980 27333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:15.191197 27333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:15.191817 27333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:15.192698 27333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:15.192755 27333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:15.192812 27333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:15.192842 27333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:15.201083 27333 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:37343
I20260929 18:46:15.201119 27461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:37343 every 8 connection(s)
I20260929 18:46:15.201977 27333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:46:15.211001 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 27333
I20260929 18:46:15.211174 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
Sep 29 18:46:15 dist-test-slave-smpn krb5kdc[26958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707575, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.67@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:15.211473 7797 external_mini_cluster.cc:1428] Setting key 32110e672ce6befae9f1c0ba1cccebc3
I20260929 18:46:15.216580 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.203791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:33457 (local address 127.7.157.126:33349)
0929 18:46:15.203918 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:15.203922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:15.204810 (+ 888us) server_negotiation.cc:407] Connection header received
0929 18:46:15.205711 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:15.205714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:15.205771 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:15.205846 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:15.207028 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.207635 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.208893 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.209125 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.209217 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.211890 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:15.211893 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:15.211896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:15.211926 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:15.213762 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:15.214433 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:15.214444 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:15.214446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:15.214555 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:15.215095 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:15.215099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:15.215101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:15.215342 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:15.215450 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:15.216271 (+ 821us) server_negotiation.cc:299] Negotiation successful
0929 18:46:15.216408 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:46:15.217783 27464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.204197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:33349 (local address 127.7.157.67:33457)
0929 18:46:15.204663 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:15.204708 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:15.205507 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:15.206031 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:15.206038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:15.206389 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:15.206884 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:15.206897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.207759 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.207762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:15.208785 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:15.208795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.209274 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.209277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:15.209465 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.210012 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:15.210029 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:15.211714 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:15.213908 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:15.213915 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:15.213934 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:15.214321 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:15.214653 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:15.214656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:15.214658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:15.214920 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:15.215457 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:15.215464 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:15.215852 (+ 388us) client_negotiation.cc:769] Sending connection context
0929 18:46:15.216199 (+ 347us) client_negotiation.cc:240] Negotiation successful
0929 18:46:15.216613 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":117,"threads_started":1}
I20260929 18:46:15.220000 27462 heartbeater.cc:347] Connected to a master server at 127.7.157.126:33349
I20260929 18:46:15.220248 27462 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:15.221029 27462 heartbeater.cc:511] Master 127.7.157.126:33349 requested a full tablet report, sending...
I20260929 18:46:15.222321 27003 ts_manager.cc:194] Registered new tserver with Master: d4aca2ab7672433984d09a29bead7f0a (127.7.157.67:37343)
I20260929 18:46:15.223141 27003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.157.67@KRBTEST.COM'} at 127.7.157.67:33457
I20260929 18:46:15.226788 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:46:15.237931 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.228947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.7.157.126:33349)
0929 18:46:15.229096 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:15.229101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:15.229356 (+ 255us) server_negotiation.cc:407] Connection header received
0929 18:46:15.229492 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:15.229494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:15.229540 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:15.229603 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:15.230464 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.231312 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.232462 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.232804 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.232906 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.234485 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:15.234490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:15.234492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:15.234519 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:15.236070 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:15.236548 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:15.236552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:15.236554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:15.236647 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:15.236957 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:15.236960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:15.236962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:15.237250 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:15.237389 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:15.237666 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 18:46:15.237778 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 18:46:15.241078 27003 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:46:15.244540 27003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:46:15.259483 27476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.253791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.7.157.67:37343)
0929 18:46:15.254234 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:15.254246 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:15.254544 (+ 298us) server_negotiation.cc:407] Connection header received
0929 18:46:15.254785 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:15.254791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:15.255003 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:15.255160 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:15.255733 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.256890 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.258210 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.258733 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.258825 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.258971 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:15.259203 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 18:46:15.259302 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":91,"threads_started":1}
I20260929 18:46:15.259473 27474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.253484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:35129 (local address 127.0.0.1:44018)
0929 18:46:15.254399 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:15.254474 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:15.254603 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:15.255481 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:15.255483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:15.255497 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:15.255658 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:15.255665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.256724 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.256728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:15.257612 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:15.257621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.258848 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.258851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:15.259029 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.259052 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:46:15.259170 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 18:46:15.259316 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":111,"thread_start_us":83,"threads_started":1}
I20260929 18:46:15.259845 27475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.253748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.7.157.66:35129)
0929 18:46:15.254886 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:15.254897 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:15.254919 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:46:15.254989 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:15.254994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:15.255212 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:15.255339 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:15.255776 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.256602 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.257726 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.258560 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.258657 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.258790 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:15.259664 (+ 874us) server_negotiation.cc:299] Negotiation successful
0929 18:46:15.259743 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1041,"thread_start_us":96,"threads_started":1}
I20260929 18:46:15.259475 27473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.253571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:37343 (local address 127.0.0.1:41074)
0929 18:46:15.254398 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:15.254460 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:15.254615 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:15.255372 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:15.255377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:15.255401 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:15.255598 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:15.255606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.257041 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.257044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:15.258106 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:15.258115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.258846 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.258849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:15.258968 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.259047 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 18:46:15.259173 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 18:46:15.259330 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":53,"thread_start_us":112,"threads_started":1}
I20260929 18:46:15.260303 27477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.254586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:34195 (local address 127.0.0.1:49120)
0929 18:46:15.255137 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:15.255150 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:15.255227 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:15.255746 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:15.255749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:15.255766 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:15.255834 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:15.255839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.257849 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.257852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:15.258897 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:15.258905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.259958 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.259961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:15.260040 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.260062 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:46:15.260125 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:46:15.260192 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":71,"threads_started":1}
I20260929 18:46:15.261077 27478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.254980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.7.157.65:34195)
0929 18:46:15.255359 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:15.255374 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:15.255392 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:46:15.255457 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:15.255461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:15.255624 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:15.255759 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:15.255916 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.257722 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.259025 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.259834 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.260009 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.260147 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:15.260194 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 18:46:15.260967 (+ 773us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":115,"threads_started":1}
I20260929 18:46:15.263253 27128 tablet_service.cc:1467] Processing CreateTablet for tablet aa58cf1987394bf5a7b45c34f51e2c47 (DEFAULT_TABLE table=test-table [id=34b8ec49eafc43898b35f7922a4a3eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:15.263110 27396 tablet_service.cc:1467] Processing CreateTablet for tablet aa58cf1987394bf5a7b45c34f51e2c47 (DEFAULT_TABLE table=test-table [id=34b8ec49eafc43898b35f7922a4a3eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:15.263128 27262 tablet_service.cc:1467] Processing CreateTablet for tablet aa58cf1987394bf5a7b45c34f51e2c47 (DEFAULT_TABLE table=test-table [id=34b8ec49eafc43898b35f7922a4a3eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:15.264209 27262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa58cf1987394bf5a7b45c34f51e2c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:15.264210 27128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa58cf1987394bf5a7b45c34f51e2c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:15.264214 27396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa58cf1987394bf5a7b45c34f51e2c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:15.270570 27479 tablet_bootstrap.cc:492] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b: Bootstrap starting.
I20260929 18:46:15.273000 27479 tablet_bootstrap.cc:654] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:15.273980 27479 log.cc:824] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:15.274667 27480 tablet_bootstrap.cc:492] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c: Bootstrap starting.
I20260929 18:46:15.275977 27479 tablet_bootstrap.cc:492] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b: No bootstrap required, opened a new log
I20260929 18:46:15.276167 27479 ts_tablet_manager.cc:1397] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:46:15.276937 27480 tablet_bootstrap.cc:654] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:15.277619 27480 log.cc:824] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:15.278251 27481 tablet_bootstrap.cc:492] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a: Bootstrap starting.
I20260929 18:46:15.279695 27480 tablet_bootstrap.cc:492] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c: No bootstrap required, opened a new log
I20260929 18:46:15.279489 27479 raft_consensus.cc:348] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd87e01636e74d1cbffdbfb749e50b8c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35129 } } peers { permanent_uuid: "d4aca2ab7672433984d09a29bead7f0a" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37343 } } peers { permanent_uuid: "3845db93e2be4479b66ea2b11e4a432b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34195 } }
I20260929 18:46:15.279769 27479 raft_consensus.cc:374] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:15.279824 27479 raft_consensus.cc:729] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3845db93e2be4479b66ea2b11e4a432b, State: Initialized, Role: FOLLOWER
I20260929 18:46:15.279894 27480 ts_tablet_manager.cc:1397] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:46:15.280166 27481 tablet_bootstrap.cc:654] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:15.280294 27479 consensus_queue.cc:260] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd87e01636e74d1cbffdbfb749e50b8c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35129 } } peers { permanent_uuid: "d4aca2ab7672433984d09a29bead7f0a" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37343 } } peers { permanent_uuid: "3845db93e2be4479b66ea2b11e4a432b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34195 } }
I20260929 18:46:15.281044 27194 heartbeater.cc:503] Master 127.7.157.126:33349 was elected leader, sending a full tablet report...
I20260929 18:46:15.281297 27481 log.cc:824] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:15.282754 27479 ts_tablet_manager.cc:1428] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:46:15.284129 27481 tablet_bootstrap.cc:492] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a: No bootstrap required, opened a new log
I20260929 18:46:15.284057 27480 raft_consensus.cc:348] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd87e01636e74d1cbffdbfb749e50b8c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35129 } } peers { permanent_uuid: "d4aca2ab7672433984d09a29bead7f0a" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37343 } } peers { permanent_uuid: "3845db93e2be4479b66ea2b11e4a432b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34195 } }
I20260929 18:46:15.284339 27481 ts_tablet_manager.cc:1397] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:46:15.284353 27480 raft_consensus.cc:374] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:15.284407 27480 raft_consensus.cc:729] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd87e01636e74d1cbffdbfb749e50b8c, State: Initialized, Role: FOLLOWER
I20260929 18:46:15.284868 27480 consensus_queue.cc:260] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd87e01636e74d1cbffdbfb749e50b8c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35129 } } peers { permanent_uuid: "d4aca2ab7672433984d09a29bead7f0a" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37343 } } peers { permanent_uuid: "3845db93e2be4479b66ea2b11e4a432b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34195 } }
I20260929 18:46:15.285571 27328 heartbeater.cc:503] Master 127.7.157.126:33349 was elected leader, sending a full tablet report...
I20260929 18:46:15.285926 27480 ts_tablet_manager.cc:1428] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:46:15.288028 27481 raft_consensus.cc:348] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd87e01636e74d1cbffdbfb749e50b8c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35129 } } peers { permanent_uuid: "d4aca2ab7672433984d09a29bead7f0a" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37343 } } peers { permanent_uuid: "3845db93e2be4479b66ea2b11e4a432b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34195 } }
I20260929 18:46:15.288311 27481 raft_consensus.cc:374] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:15.288379 27481 raft_consensus.cc:729] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4aca2ab7672433984d09a29bead7f0a, State: Initialized, Role: FOLLOWER
I20260929 18:46:15.288815 27481 consensus_queue.cc:260] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd87e01636e74d1cbffdbfb749e50b8c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35129 } } peers { permanent_uuid: "d4aca2ab7672433984d09a29bead7f0a" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37343 } } peers { permanent_uuid: "3845db93e2be4479b66ea2b11e4a432b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34195 } }
I20260929 18:46:15.289583 27462 heartbeater.cc:503] Master 127.7.157.126:33349 was elected leader, sending a full tablet report...
I20260929 18:46:15.289820 27481 ts_tablet_manager.cc:1428] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:46:15.376403 27486 raft_consensus.cc:482] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:46:15.376600 27486 raft_consensus.cc:504] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd87e01636e74d1cbffdbfb749e50b8c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35129 } } peers { permanent_uuid: "d4aca2ab7672433984d09a29bead7f0a" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37343 } } peers { permanent_uuid: "3845db93e2be4479b66ea2b11e4a432b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34195 } }
I20260929 18:46:15.377696 27486 leader_election.cc:288] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4aca2ab7672433984d09a29bead7f0a (127.7.157.67:37343), 3845db93e2be4479b66ea2b11e4a432b (127.7.157.65:34195)
I20260929 18:46:15.381263 27476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.378053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:53889 (local address 127.7.157.67:37343)
0929 18:46:15.378190 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:15.378194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:15.378294 (+ 100us) server_negotiation.cc:407] Connection header received
0929 18:46:15.378412 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:15.378414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:15.378465 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:15.378535 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:15.378796 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.379312 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.380322 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.380736 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.380847 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.380901 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:15.381074 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 18:46:15.381140 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:46:15.381263 27330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.377945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:37343 (local address 127.7.157.66:53889)
0929 18:46:15.378189 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:15.378205 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:15.378317 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:15.378569 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:15.378572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:15.378590 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:15.378695 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:15.378699 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.379417 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.379420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:15.380198 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:15.380207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.380847 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.380850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:15.380965 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.380983 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:46:15.381046 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:46:15.381123 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260929 18:46:15.381348 27478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.378052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:38663 (local address 127.7.157.65:34195)
0929 18:46:15.378202 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:15.378207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:15.378477 (+ 270us) server_negotiation.cc:407] Connection header received
0929 18:46:15.378564 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:15.378567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:15.378615 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:15.378675 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:15.378919 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.379451 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.380439 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.380828 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.380932 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.380981 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:15.381154 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 18:46:15.381226 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:46:15.381397 27488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.377945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:34195 (local address 127.7.157.66:38663)
0929 18:46:15.378383 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:15.378395 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:15.378484 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:15.378695 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:15.378698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:15.378712 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:15.378821 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:15.378826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.379549 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.379552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:15.380315 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:15.380324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.380944 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.380948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:15.381045 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.381063 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:46:15.381127 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:46:15.381195 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":70,"thread_start_us":113,"threads_started":1}
I20260929 18:46:15.381950 27148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa58cf1987394bf5a7b45c34f51e2c47" candidate_uuid: "bd87e01636e74d1cbffdbfb749e50b8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3845db93e2be4479b66ea2b11e4a432b" is_pre_election: true
I20260929 18:46:15.381959 27416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa58cf1987394bf5a7b45c34f51e2c47" candidate_uuid: "bd87e01636e74d1cbffdbfb749e50b8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4aca2ab7672433984d09a29bead7f0a" is_pre_election: true
I20260929 18:46:15.382283 27416 raft_consensus.cc:2454] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd87e01636e74d1cbffdbfb749e50b8c in term 0.
I20260929 18:46:15.382283 27148 raft_consensus.cc:2454] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd87e01636e74d1cbffdbfb749e50b8c in term 0.
I20260929 18:46:15.382761 27218 leader_election.cc:302] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3845db93e2be4479b66ea2b11e4a432b, bd87e01636e74d1cbffdbfb749e50b8c; no voters:
I20260929 18:46:15.383014 27486 raft_consensus.cc:2781] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:46:15.383113 27486 raft_consensus.cc:482] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:46:15.383160 27486 raft_consensus.cc:3037] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:15.384163 27486 raft_consensus.cc:504] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd87e01636e74d1cbffdbfb749e50b8c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35129 } } peers { permanent_uuid: "d4aca2ab7672433984d09a29bead7f0a" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37343 } } peers { permanent_uuid: "3845db93e2be4479b66ea2b11e4a432b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34195 } }
I20260929 18:46:15.384509 27486 leader_election.cc:288] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [CANDIDATE]: Term 1 election: Requested vote from peers d4aca2ab7672433984d09a29bead7f0a (127.7.157.67:37343), 3845db93e2be4479b66ea2b11e4a432b (127.7.157.65:34195)
I20260929 18:46:15.384845 27148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa58cf1987394bf5a7b45c34f51e2c47" candidate_uuid: "bd87e01636e74d1cbffdbfb749e50b8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3845db93e2be4479b66ea2b11e4a432b"
I20260929 18:46:15.384855 27416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa58cf1987394bf5a7b45c34f51e2c47" candidate_uuid: "bd87e01636e74d1cbffdbfb749e50b8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4aca2ab7672433984d09a29bead7f0a"
I20260929 18:46:15.384972 27416 raft_consensus.cc:3037] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:15.384977 27148 raft_consensus.cc:3037] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:15.385964 27416 raft_consensus.cc:2454] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd87e01636e74d1cbffdbfb749e50b8c in term 1.
I20260929 18:46:15.385983 27148 raft_consensus.cc:2454] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd87e01636e74d1cbffdbfb749e50b8c in term 1.
I20260929 18:46:15.386264 27216 leader_election.cc:302] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3845db93e2be4479b66ea2b11e4a432b, bd87e01636e74d1cbffdbfb749e50b8c, d4aca2ab7672433984d09a29bead7f0a; no voters:
I20260929 18:46:15.386478 27486 raft_consensus.cc:2781] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:15.386655 27486 raft_consensus.cc:686] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [term 1 LEADER]: Becoming Leader. State: Replica: bd87e01636e74d1cbffdbfb749e50b8c, State: Running, Role: LEADER
I20260929 18:46:15.386955 27486 consensus_queue.cc:237] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd87e01636e74d1cbffdbfb749e50b8c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35129 } } peers { permanent_uuid: "d4aca2ab7672433984d09a29bead7f0a" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37343 } } peers { permanent_uuid: "3845db93e2be4479b66ea2b11e4a432b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34195 } }
I20260929 18:46:15.390108 27003 catalog_manager.cc:5495] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c reported cstate change: term changed from 0 to 1, leader changed from <none> to bd87e01636e74d1cbffdbfb749e50b8c (127.7.157.66). New cstate: current_term: 1 leader_uuid: "bd87e01636e74d1cbffdbfb749e50b8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd87e01636e74d1cbffdbfb749e50b8c" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4aca2ab7672433984d09a29bead7f0a" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3845db93e2be4479b66ea2b11e4a432b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34195 } health_report { overall_health: UNKNOWN } } }
I20260929 18:46:15.410015 27475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.406427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.7.157.66:35129)
0929 18:46:15.406548 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:15.406552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:15.406564 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:46:15.406697 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:15.406699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:15.406752 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:15.406836 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:46:15.407155 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.407641 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.408633 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.408839 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.408921 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.409216 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:46:15.409621 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:46:15.409725 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:15.409798 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:46:15.409855 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:46:15.415575 27148 raft_consensus.cc:1261] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b [term 1 FOLLOWER]: Refusing update from remote peer bd87e01636e74d1cbffdbfb749e50b8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:46:15.415575 27416 raft_consensus.cc:1261] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a [term 1 FOLLOWER]: Refusing update from remote peer bd87e01636e74d1cbffdbfb749e50b8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:46:15.416236 27489 consensus_queue.cc:1035] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4aca2ab7672433984d09a29bead7f0a" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 37343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:46:15.416419 27486 consensus_queue.cc:1035] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3845db93e2be4479b66ea2b11e4a432b" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 34195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:46:15.424108 27492 mvcc.cc:205] Tried to move back new op lower bound from 7334738228895137792 to 7334738228787662848. Current Snapshot: MvccSnapshot[applied={T|T < 7334738228895137792}]
I20260929 18:46:15.424343 27493 mvcc.cc:205] Tried to move back new op lower bound from 7334738228895137792 to 7334738228787662848. Current Snapshot: MvccSnapshot[applied={T|T < 7334738228895137792}]
I20260929 18:46:15.433502 27003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:46:15.433662 27003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:46:15.436048 27003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d49ce6db91846b1b63bd95a0c94e195: Sending DeleteTablet for 3 replicas of tablet aa58cf1987394bf5a7b45c34f51e2c47
I20260929 18:46:15.436843 27262 tablet_service.cc:1514] Processing DeleteTablet for tablet aa58cf1987394bf5a7b45c34f51e2c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:15 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:44018
I20260929 18:46:15.436872 27128 tablet_service.cc:1514] Processing DeleteTablet for tablet aa58cf1987394bf5a7b45c34f51e2c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:15 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:49120
I20260929 18:46:15.436846 27396 tablet_service.cc:1514] Processing DeleteTablet for tablet aa58cf1987394bf5a7b45c34f51e2c47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:15 UTC) from {username='kudu', principal='kudu/127.7.157.126@KRBTEST.COM'} at 127.0.0.1:41074
I20260929 18:46:15.437337 27504 tablet_replica.cc:330] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b: stopping tablet replica
I20260929 18:46:15.437381 27502 tablet_replica.cc:330] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c: stopping tablet replica
I20260929 18:46:15.437659 27504 raft_consensus.cc:2229] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:46:15.437700 27502 raft_consensus.cc:2229] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:46:15.437989 27504 raft_consensus.cc:2258] T aa58cf1987394bf5a7b45c34f51e2c47 P 3845db93e2be4479b66ea2b11e4a432b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:46:15.438045 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 27065
I20260929 18:46:15.438297 27502 raft_consensus.cc:2258] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:46:15.438359 27503 tablet_replica.cc:330] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a: stopping tablet replica
I20260929 18:46:15.438614 27503 raft_consensus.cc:2229] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:46:15.439184 27503 raft_consensus.cc:2258] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:46:15.440493 27502 ts_tablet_manager.cc:1905] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:46:15.442685 27502 ts_tablet_manager.cc:1918] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:46:15.442775 27502 log.cc:1196] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-1/wal/wals/aa58cf1987394bf5a7b45c34f51e2c47
I20260929 18:46:15.443038 27502 ts_tablet_manager.cc:1939] T aa58cf1987394bf5a7b45c34f51e2c47 P bd87e01636e74d1cbffdbfb749e50b8c: Deleting consensus metadata
I20260929 18:46:15.444309 26990 catalog_manager.cc:4841] TS bd87e01636e74d1cbffdbfb749e50b8c (127.7.157.66:35129): tablet aa58cf1987394bf5a7b45c34f51e2c47 (table test-table [id=34b8ec49eafc43898b35f7922a4a3eec]) successfully deleted
I20260929 18:46:15.444497 27503 ts_tablet_manager.cc:1905] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:46:15.446161 26990 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:34195 (error 108)
I20260929 18:46:15.446861 27503 ts_tablet_manager.cc:1918] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:46:15.446961 27503 log.cc:1196] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/aa58cf1987394bf5a7b45c34f51e2c47
I20260929 18:46:15.447263 27503 ts_tablet_manager.cc:1939] T aa58cf1987394bf5a7b45c34f51e2c47 P d4aca2ab7672433984d09a29bead7f0a: Deleting consensus metadata
I20260929 18:46:15.447274 27477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.446922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:34195 (local address 127.0.0.1:49136)
0929 18:46:15.447120 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:15.447186 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:34195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260929 18:46:15.447425 26990 catalog_manager.cc:4584] TS 3845db93e2be4479b66ea2b11e4a432b (127.7.157.65:34195): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa58cf1987394bf5a7b45c34f51e2c47: Network error: Client connection negotiation failed: client connection to 127.7.157.65:34195: connect: Connection refused (error 111)
I20260929 18:46:15.447834 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 27199
I20260929 18:46:15.449023 26988 catalog_manager.cc:4841] TS d4aca2ab7672433984d09a29bead7f0a (127.7.157.67:37343): tablet aa58cf1987394bf5a7b45c34f51e2c47 (table test-table [id=34b8ec49eafc43898b35f7922a4a3eec]) successfully deleted
I20260929 18:46:15.452924 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 27333
I20260929 18:46:15.457579 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 26971
2026-09-29T18:46:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2214 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T18:46:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:15Z Disabled control of system clock
I20260929 18:46:15.484881 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:33877
--webserver_interface=127.7.157.126
--webserver_port=0
--builtin_ntp_servers=127.7.157.84:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:33877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:46:15.591002 27509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:15.591281 27509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:15.591315 27509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:15.595280 27509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:15.595351 27509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:15.595377 27509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:15.595398 27509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:15.595427 27509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:15.600191 27509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:33877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:33877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:15.601253 27509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:15.602625 27509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:15.610729 27515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:15.610729 27517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:15.610743 27514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:15.611097 27509 server_base.cc:1034] running on GCE node
I20260929 18:46:15.611646 27509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:15.612700 27509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:15.613847 27509 hybrid_clock.cc:648] HybridClock initialized: now 1790707575613832 us; error 29 us; skew 500 ppm
I20260929 18:46:15.615587 27509 webserver.cc:460] Webserver started at http://127.7.157.126:40845/ using document root <none> and password file <none>
I20260929 18:46:15.616117 27509 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:15.616174 27509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:15.616364 27509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:15.618279 27509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "94b27b2f08644048a358fd7f83c0e8ef"
format_stamp: "Formatted at 2026-09-29 18:46:15 on dist-test-slave-smpn"
server_key: "6017d8f4d5207db04f1a1a9516e0d1cc"
server_key_iv: "36bb22043b8f63e3ffc7fb74d47d0ada"
server_key_version: "encryptionkey@0"
I20260929 18:46:15.618781 27509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "94b27b2f08644048a358fd7f83c0e8ef"
format_stamp: "Formatted at 2026-09-29 18:46:15 on dist-test-slave-smpn"
server_key: "6017d8f4d5207db04f1a1a9516e0d1cc"
server_key_iv: "36bb22043b8f63e3ffc7fb74d47d0ada"
server_key_version: "encryptionkey@0"
I20260929 18:46:15.622207 27509 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:46:15.624408 27523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:15.625517 27509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:46:15.625622 27509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "94b27b2f08644048a358fd7f83c0e8ef"
format_stamp: "Formatted at 2026-09-29 18:46:15 on dist-test-slave-smpn"
server_key: "6017d8f4d5207db04f1a1a9516e0d1cc"
server_key_iv: "36bb22043b8f63e3ffc7fb74d47d0ada"
server_key_version: "encryptionkey@0"
I20260929 18:46:15.625706 27509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:15.653767 27509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:15.654513 27509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:15.654698 27509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:15.661523 27509 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:33877
I20260929 18:46:15.661540 27575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:33877 every 8 connection(s)
I20260929 18:46:15.662497 27509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:15.665596 27576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:15.669703 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 27509
I20260929 18:46:15.669811 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:15.670051 7797 external_mini_cluster.cc:1428] Setting key 4a3df2deff0a579a653030bf3ccafbe6
I20260929 18:46:15.671424 27576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef: Bootstrap starting.
I20260929 18:46:15.674196 27576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:15.675159 27576 log.cc:824] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:15.677188 27576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef: No bootstrap required, opened a new log
I20260929 18:46:15.680440 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.671478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.7.157.126:33877)
0929 18:46:15.672166 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:15.672190 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:15.672233 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:46:15.673092 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:15.673118 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:15.673637 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:15.674022 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:15.674507 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.675857 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.676742 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.677445 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.677611 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.678342 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:15.678348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:15.678364 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:15.678859 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:15.678929 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:15.678967 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:15.679102 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:15.679186 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 18:46:15.679467 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":108,"threads_started":1}
I20260929 18:46:15.681608 27576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b27b2f08644048a358fd7f83c0e8ef" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33877 } }
I20260929 18:46:15.681881 27576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:15.681939 27576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b27b2f08644048a358fd7f83c0e8ef, State: Initialized, Role: FOLLOWER
I20260929 18:46:15.682395 27576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b27b2f08644048a358fd7f83c0e8ef" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33877 } }
I20260929 18:46:15.682564 27576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:15.682633 27576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:15.682731 27576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:15.683693 27576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b27b2f08644048a358fd7f83c0e8ef" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33877 } }
I20260929 18:46:15.684018 27576 leader_election.cc:302] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94b27b2f08644048a358fd7f83c0e8ef; no voters:
I20260929 18:46:15.684348 27576 leader_election.cc:288] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:15.684459 27581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:15.684651 27581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [term 1 LEADER]: Becoming Leader. State: Replica: 94b27b2f08644048a358fd7f83c0e8ef, State: Running, Role: LEADER
I20260929 18:46:15.684959 27581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b27b2f08644048a358fd7f83c0e8ef" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33877 } }
I20260929 18:46:15.685586 27576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:15.686556 27582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94b27b2f08644048a358fd7f83c0e8ef. Latest consensus state: current_term: 1 leader_uuid: "94b27b2f08644048a358fd7f83c0e8ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b27b2f08644048a358fd7f83c0e8ef" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33877 } } }
I20260929 18:46:15.686586 27583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94b27b2f08644048a358fd7f83c0e8ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b27b2f08644048a358fd7f83c0e8ef" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33877 } } }
I20260929 18:46:15.686688 27582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:15.686691 27583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:15.687095 27590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:15.689778 27590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:15.695036 27590 catalog_manager.cc:1347] Generated new cluster ID: f447b012998b424cbdbb80c7cf6d29c0
I20260929 18:46:15.695142 27590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:15.720255 27590 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:15.721585 27590 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:15.726023 27590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef: Generated new TSK 0
I20260929 18:46:15.726838 27590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:15.732026 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.65:0
--local_ip_for_outbound_sockets=127.7.157.65
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:33877
--builtin_ntp_servers=127.7.157.84:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:46:15.842983 27600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:15.843307 27600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:15.843361 27600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:15.847342 27600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:15.847395 27600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:15.847466 27600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.65
I20260929 18:46:15.852219 27600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.157.65
--webserver_port=0
--tserver_master_addrs=127.7.157.126:33877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.65
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:15.853305 27600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:15.854673 27600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:46:15.864346 27600 server_base.cc:1034] running on GCE node
W20260929 18:46:15.864303 27606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:15.864295 27608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:15.864305 27605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:15.865000 27600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:15.865676 27600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:15.866833 27600 hybrid_clock.cc:648] HybridClock initialized: now 1790707575866807 us; error 37 us; skew 500 ppm
I20260929 18:46:15.868664 27600 webserver.cc:460] Webserver started at http://127.7.157.65:36569/ using document root <none> and password file <none>
I20260929 18:46:15.869220 27600 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:15.869282 27600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:15.869486 27600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:15.871567 27600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/data/instance:
uuid: "6977d98d07a64e6aa7117f874334f86a"
format_stamp: "Formatted at 2026-09-29 18:46:15 on dist-test-slave-smpn"
server_key: "2379f78ded84cf4b724737c95e93c26a"
server_key_iv: "93aebf8a25d19d24cbfd0bd25d1a86ad"
server_key_version: "encryptionkey@0"
I20260929 18:46:15.872089 27600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance:
uuid: "6977d98d07a64e6aa7117f874334f86a"
format_stamp: "Formatted at 2026-09-29 18:46:15 on dist-test-slave-smpn"
server_key: "2379f78ded84cf4b724737c95e93c26a"
server_key_iv: "93aebf8a25d19d24cbfd0bd25d1a86ad"
server_key_version: "encryptionkey@0"
I20260929 18:46:15.875649 27600 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:46:15.878067 27614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:15.879285 27600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:15.879380 27600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/wal
uuid: "6977d98d07a64e6aa7117f874334f86a"
format_stamp: "Formatted at 2026-09-29 18:46:15 on dist-test-slave-smpn"
server_key: "2379f78ded84cf4b724737c95e93c26a"
server_key_iv: "93aebf8a25d19d24cbfd0bd25d1a86ad"
server_key_version: "encryptionkey@0"
I20260929 18:46:15.879484 27600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:15.901147 27600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:15.901873 27600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:15.902060 27600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:15.902665 27600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:15.903687 27600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:15.903733 27600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:15.903780 27600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:15.903810 27600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:15.912412 27600 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.65:42803
I20260929 18:46:15.912456 27727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.65:42803 every 8 connection(s)
I20260929 18:46:15.913354 27600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/data/info.pb
I20260929 18:46:15.918140 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 27600
I20260929 18:46:15.918249 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-0/wal/instance
I20260929 18:46:15.918509 7797 external_mini_cluster.cc:1428] Setting key 0953dda7c7aee561586d1de374b9e840
I20260929 18:46:15.920473 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.66:0
--local_ip_for_outbound_sockets=127.7.157.66
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:33877
--builtin_ntp_servers=127.7.157.84:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:46:15.923223 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.915202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:43357 (local address 127.7.157.126:33877)
0929 18:46:15.915343 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:15.915347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:15.916499 (+ 1152us) server_negotiation.cc:407] Connection header received
0929 18:46:15.917908 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:15.917912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:15.917976 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:15.918049 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:15.919131 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.919761 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.921077 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.921394 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.921516 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.922715 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:15.922718 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:15.922720 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:15.922732 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:15.922754 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:15.922762 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:15.922819 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:15.922993 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 18:46:15.923053 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:46:15.924085 27730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:15.915649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:33877 (local address 127.7.157.65:43357)
0929 18:46:15.916333 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:15.916376 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:15.917660 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:15.918257 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:15.918266 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:15.918374 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:15.918963 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:15.918982 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.919902 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.919905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:15.920959 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:15.920969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:15.921555 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:15.921557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:15.921764 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:15.922405 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:46:15.922408 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:15.922468 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:46:15.922555 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:15.922837 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:15.922844 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:15.922849 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:46:15.922972 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 18:46:15.923247 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260929 18:46:15.925377 27728 heartbeater.cc:347] Connected to a master server at 127.7.157.126:33877
I20260929 18:46:15.925611 27728 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:15.926128 27728 heartbeater.cc:511] Master 127.7.157.126:33877 requested a full tablet report, sending...
I20260929 18:46:15.927680 27540 ts_manager.cc:194] Registered new tserver with Master: 6977d98d07a64e6aa7117f874334f86a (127.7.157.65:42803)
I20260929 18:46:15.928859 27540 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.157.65:43357
W20260929 18:46:16.030826 27731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:16.031113 27731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:16.031153 27731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:16.035181 27731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:16.035238 27731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:16.035305 27731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.66
I20260929 18:46:16.040205 27731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.157.66
--webserver_port=0
--tserver_master_addrs=127.7.157.126:33877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.66
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:16.041316 27731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:16.042683 27731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:16.051991 27736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:16.051982 27739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:16.051980 27737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:16.052241 27731 server_base.cc:1034] running on GCE node
I20260929 18:46:16.052714 27731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:16.053354 27731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:16.054531 27731 hybrid_clock.cc:648] HybridClock initialized: now 1790707576054504 us; error 42 us; skew 500 ppm
I20260929 18:46:16.056425 27731 webserver.cc:460] Webserver started at http://127.7.157.66:44839/ using document root <none> and password file <none>
I20260929 18:46:16.056972 27731 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:16.057034 27731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:16.057232 27731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:16.059330 27731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/data/instance:
uuid: "b53401294ca44589baa9023466f9cf36"
format_stamp: "Formatted at 2026-09-29 18:46:16 on dist-test-slave-smpn"
server_key: "9e20cf418f7948527ab2aaeb926ae1ee"
server_key_iv: "07eb86de167e7cb9925f287b96c5f3ce"
server_key_version: "encryptionkey@0"
I20260929 18:46:16.059870 27731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance:
uuid: "b53401294ca44589baa9023466f9cf36"
format_stamp: "Formatted at 2026-09-29 18:46:16 on dist-test-slave-smpn"
server_key: "9e20cf418f7948527ab2aaeb926ae1ee"
server_key_iv: "07eb86de167e7cb9925f287b96c5f3ce"
server_key_version: "encryptionkey@0"
I20260929 18:46:16.063562 27731 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:46:16.066007 27745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:16.067345 27731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:16.067456 27731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/wal
uuid: "b53401294ca44589baa9023466f9cf36"
format_stamp: "Formatted at 2026-09-29 18:46:16 on dist-test-slave-smpn"
server_key: "9e20cf418f7948527ab2aaeb926ae1ee"
server_key_iv: "07eb86de167e7cb9925f287b96c5f3ce"
server_key_version: "encryptionkey@0"
I20260929 18:46:16.067548 27731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:16.092752 27731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:16.093493 27731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:16.093676 27731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:16.094285 27731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:16.095289 27731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:16.095338 27731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:16.095394 27731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:16.095420 27731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:16.103757 27731 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.66:35399
I20260929 18:46:16.103791 27858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.66:35399 every 8 connection(s)
I20260929 18:46:16.104646 27731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/data/info.pb
I20260929 18:46:16.107483 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 27731
I20260929 18:46:16.107605 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-1/wal/instance
I20260929 18:46:16.107893 7797 external_mini_cluster.cc:1428] Setting key b40ae56ba5536278509880c1b840cbc4
I20260929 18:46:16.109917 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.157.67:0
--local_ip_for_outbound_sockets=127.7.157.67
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:33877
--builtin_ntp_servers=127.7.157.84:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:46:16.114677 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.106356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:37979 (local address 127.7.157.126:33877)
0929 18:46:16.106525 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:16.106531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:16.107625 (+ 1094us) server_negotiation.cc:407] Connection header received
0929 18:46:16.109210 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:16.109213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:16.109263 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:16.109328 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:16.110325 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.111051 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.112665 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.112878 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.112952 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.114230 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:16.114234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:16.114236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:16.114249 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:16.114270 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:16.114278 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:16.114334 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:16.114501 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 18:46:16.114558 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:46:16.115976 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.106803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:33877 (local address 127.7.157.66:37979)
0929 18:46:16.107488 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:16.107527 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:16.108993 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:16.109524 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:16.109531 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:16.109622 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:16.110190 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.110206 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.111205 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.111208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.112544 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.112556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.113020 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.113023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.113224 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.113897 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:46:16.113899 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:16.113973 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:46:16.114067 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:16.114355 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:16.114364 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:16.114370 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:46:16.114501 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 18:46:16.114838 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":124,"threads_started":1}
I20260929 18:46:16.117173 27859 heartbeater.cc:347] Connected to a master server at 127.7.157.126:33877
I20260929 18:46:16.117408 27859 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:16.117955 27859 heartbeater.cc:511] Master 127.7.157.126:33877 requested a full tablet report, sending...
I20260929 18:46:16.119056 27540 ts_manager.cc:194] Registered new tserver with Master: b53401294ca44589baa9023466f9cf36 (127.7.157.66:35399)
I20260929 18:46:16.119875 27540 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.157.66:37979
W20260929 18:46:16.219162 27862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:16.219434 27862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:16.219470 27862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:16.223739 27862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:16.223805 27862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:16.223860 27862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.157.67
I20260929 18:46:16.228927 27862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.157.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.157.67
--webserver_port=0
--tserver_master_addrs=127.7.157.126:33877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.157.67
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:16.230129 27862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:16.231561 27862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:46:16.240829 27862 server_base.cc:1034] running on GCE node
W20260929 18:46:16.240777 27867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:16.240777 27868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:16.240777 27870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:16.241405 27862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:16.242074 27862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:16.243238 27862 hybrid_clock.cc:648] HybridClock initialized: now 1790707576243201 us; error 41 us; skew 500 ppm
I20260929 18:46:16.245035 27862 webserver.cc:460] Webserver started at http://127.7.157.67:39765/ using document root <none> and password file <none>
I20260929 18:46:16.245606 27862 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:16.245667 27862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:16.245863 27862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:16.247974 27862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/data/instance:
uuid: "828d10c0e67447c28911b93b1a1da9ed"
format_stamp: "Formatted at 2026-09-29 18:46:16 on dist-test-slave-smpn"
server_key: "0612187f5868dee30bd998050aef0b74"
server_key_iv: "368f65e2aa30320f0dc5c47940cb8fef"
server_key_version: "encryptionkey@0"
I20260929 18:46:16.248525 27862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance:
uuid: "828d10c0e67447c28911b93b1a1da9ed"
format_stamp: "Formatted at 2026-09-29 18:46:16 on dist-test-slave-smpn"
server_key: "0612187f5868dee30bd998050aef0b74"
server_key_iv: "368f65e2aa30320f0dc5c47940cb8fef"
server_key_version: "encryptionkey@0"
I20260929 18:46:16.252095 27862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260929 18:46:16.254499 27876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:16.255759 27862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:16.255856 27862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/wal
uuid: "828d10c0e67447c28911b93b1a1da9ed"
format_stamp: "Formatted at 2026-09-29 18:46:16 on dist-test-slave-smpn"
server_key: "0612187f5868dee30bd998050aef0b74"
server_key_iv: "368f65e2aa30320f0dc5c47940cb8fef"
server_key_version: "encryptionkey@0"
I20260929 18:46:16.255964 27862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:16.267225 27862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:16.267966 27862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:16.268158 27862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:16.268844 27862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:46:16.269909 27862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:46:16.269968 27862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:16.270081 27862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:46:16.270117 27862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:16.278972 27862 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.67:40999
I20260929 18:46:16.279002 27989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.67:40999 every 8 connection(s)
I20260929 18:46:16.279947 27862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/data/info.pb
I20260929 18:46:16.286171 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 27862
I20260929 18:46:16.286291 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/wal/instance
I20260929 18:46:16.286585 7797 external_mini_cluster.cc:1428] Setting key 2c3832557242f4c921f3b22f20c5215e
I20260929 18:46:16.290386 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.281841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.67:33317 (local address 127.7.157.126:33877)
0929 18:46:16.282010 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:16.282015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:16.283726 (+ 1711us) server_negotiation.cc:407] Connection header received
0929 18:46:16.285187 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:16.285190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:16.285242 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:16.285326 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:16.286408 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.287037 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.288491 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.288789 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.288883 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.289911 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:16.289915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:16.289918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:16.289932 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:16.289962 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:16.289971 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:16.290030 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:16.290177 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 18:46:16.290241 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:46:16.291334 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.282287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:33877 (local address 127.7.157.67:33317)
0929 18:46:16.283555 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:16.283599 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:16.284954 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:16.285504 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:16.285513 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:16.285624 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:16.286259 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.286282 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.287174 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.287177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.288372 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.288383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.288946 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.288951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.289120 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.289655 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:46:16.289657 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:16.289699 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:46:16.289767 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:16.290042 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:16.290052 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:16.290058 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:46:16.290172 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 18:46:16.290490 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":136,"threads_started":1}
I20260929 18:46:16.292440 27990 heartbeater.cc:347] Connected to a master server at 127.7.157.126:33877
I20260929 18:46:16.292717 27990 heartbeater.cc:464] Registering TS with master...
I20260929 18:46:16.293381 27990 heartbeater.cc:511] Master 127.7.157.126:33877 requested a full tablet report, sending...
I20260929 18:46:16.294596 27540 ts_manager.cc:194] Registered new tserver with Master: 828d10c0e67447c28911b93b1a1da9ed (127.7.157.67:40999)
I20260929 18:46:16.295420 27540 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.157.67:33317
I20260929 18:46:16.300771 7797 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:46:16.306223 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.302928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54898 (local address 127.7.157.126:33877)
0929 18:46:16.303033 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:16.303037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:16.303248 (+ 211us) server_negotiation.cc:407] Connection header received
0929 18:46:16.303356 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:16.303358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:16.303403 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:16.303476 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:16.303689 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.304191 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.304963 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.305167 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.305234 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.305891 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:16.305894 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:16.305896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:16.305908 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:16.305928 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:16.305936 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:16.305988 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:16.306047 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 18:46:16.306102 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:46:16.309054 27540 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:46:16.312125 27540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:46:16.326584 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.321208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:42803 (local address 127.0.0.1:59702)
0929 18:46:16.321921 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:16.321933 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:16.322005 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:16.322475 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:16.322477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:16.322496 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:16.322587 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.322593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.323650 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.323653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.325232 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.325240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.326024 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.326027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.326131 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.326167 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 18:46:16.326281 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 18:46:16.326407 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":210,"thread_start_us":88,"threads_started":1}
I20260929 18:46:16.326784 28004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.321096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.7.157.67:40999)
0929 18:46:16.322032 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:16.322044 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:16.322062 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:46:16.322137 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:16.322142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:16.322334 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:16.322496 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:16.322893 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.323987 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.325213 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.326021 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.326128 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.326309 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:16.326511 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 18:46:16.326620 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":131,"threads_started":1}
I20260929 18:46:16.326619 28005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.320498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:40999 (local address 127.0.0.1:42994)
0929 18:46:16.321599 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:16.321612 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:16.321931 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:16.322651 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:16.322654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:16.322673 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:16.322769 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.322775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.324110 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.324113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.325089 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.325099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.326287 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.326291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.326373 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.326396 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:46:16.326465 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:46:16.326524 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260929 18:46:16.327008 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.321386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59702 (local address 127.7.157.65:42803)
0929 18:46:16.321830 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:16.321843 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:16.322066 (+ 223us) server_negotiation.cc:407] Connection header received
0929 18:46:16.322138 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:16.322142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:16.322334 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:16.322496 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:16.322698 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.323545 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.325352 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.325900 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.326002 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.326212 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:16.326722 (+ 510us) server_negotiation.cc:299] Negotiation successful
0929 18:46:16.326832 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":154,"threads_started":1}
I20260929 18:46:16.327284 28002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.320430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:35399 (local address 127.0.0.1:54964)
0929 18:46:16.321520 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:16.321571 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:16.321704 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:16.322241 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:16.322246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:16.322268 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:16.322403 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.322409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.323846 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.323849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.325191 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.325202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.326373 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.326376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.326464 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.326487 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:46:16.326555 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:46:16.326620 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":57,"thread_start_us":88,"threads_started":1}
I20260929 18:46:16.327665 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.320591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.7.157.66:35399)
0929 18:46:16.320952 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:16.320970 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:16.321729 (+ 759us) server_negotiation.cc:407] Connection header received
0929 18:46:16.321849 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:16.321854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:16.322100 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:16.322243 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:16.322541 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.323737 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.325315 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.326046 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.326132 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.326309 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:16.327396 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0929 18:46:16.327518 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":144,"threads_started":1}
I20260929 18:46:16.329766 27924 tablet_service.cc:1467] Processing CreateTablet for tablet 8a074f4195d94801847030a721e96de2 (DEFAULT_TABLE table=test-table [id=1277e013aeaf44fe9e7d51d5a9e5c4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:16.329807 27662 tablet_service.cc:1467] Processing CreateTablet for tablet 8a074f4195d94801847030a721e96de2 (DEFAULT_TABLE table=test-table [id=1277e013aeaf44fe9e7d51d5a9e5c4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:16.330052 27793 tablet_service.cc:1467] Processing CreateTablet for tablet 8a074f4195d94801847030a721e96de2 (DEFAULT_TABLE table=test-table [id=1277e013aeaf44fe9e7d51d5a9e5c4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:46:16.330708 27924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a074f4195d94801847030a721e96de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:16.330828 27662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a074f4195d94801847030a721e96de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:16.330828 27793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a074f4195d94801847030a721e96de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:16.337319 28007 tablet_bootstrap.cc:492] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed: Bootstrap starting.
I20260929 18:46:16.338888 28009 tablet_bootstrap.cc:492] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36: Bootstrap starting.
I20260929 18:46:16.339098 28008 tablet_bootstrap.cc:492] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a: Bootstrap starting.
I20260929 18:46:16.339588 28007 tablet_bootstrap.cc:654] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:16.340288 28007 log.cc:824] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:16.341147 28008 tablet_bootstrap.cc:654] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:16.341146 28009 tablet_bootstrap.cc:654] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:16.341969 28008 log.cc:824] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:16.341971 28009 log.cc:824] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:16.342223 28007 tablet_bootstrap.cc:492] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed: No bootstrap required, opened a new log
I20260929 18:46:16.342412 28007 ts_tablet_manager.cc:1397] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:46:16.343837 28009 tablet_bootstrap.cc:492] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36: No bootstrap required, opened a new log
I20260929 18:46:16.344020 28009 ts_tablet_manager.cc:1397] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:46:16.344295 28008 tablet_bootstrap.cc:492] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a: No bootstrap required, opened a new log
I20260929 18:46:16.344485 28008 ts_tablet_manager.cc:1397] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:46:16.346619 28007 raft_consensus.cc:348] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53401294ca44589baa9023466f9cf36" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35399 } } peers { permanent_uuid: "828d10c0e67447c28911b93b1a1da9ed" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40999 } } peers { permanent_uuid: "6977d98d07a64e6aa7117f874334f86a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 42803 } }
I20260929 18:46:16.346901 28007 raft_consensus.cc:374] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:16.346966 28007 raft_consensus.cc:729] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 828d10c0e67447c28911b93b1a1da9ed, State: Initialized, Role: FOLLOWER
I20260929 18:46:16.347092 28009 raft_consensus.cc:348] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53401294ca44589baa9023466f9cf36" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35399 } } peers { permanent_uuid: "828d10c0e67447c28911b93b1a1da9ed" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40999 } } peers { permanent_uuid: "6977d98d07a64e6aa7117f874334f86a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 42803 } }
I20260929 18:46:16.347337 28009 raft_consensus.cc:374] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:16.347397 28009 raft_consensus.cc:729] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53401294ca44589baa9023466f9cf36, State: Initialized, Role: FOLLOWER
I20260929 18:46:16.347520 28007 consensus_queue.cc:260] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53401294ca44589baa9023466f9cf36" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35399 } } peers { permanent_uuid: "828d10c0e67447c28911b93b1a1da9ed" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40999 } } peers { permanent_uuid: "6977d98d07a64e6aa7117f874334f86a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 42803 } }
I20260929 18:46:16.347810 28009 consensus_queue.cc:260] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53401294ca44589baa9023466f9cf36" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35399 } } peers { permanent_uuid: "828d10c0e67447c28911b93b1a1da9ed" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40999 } } peers { permanent_uuid: "6977d98d07a64e6aa7117f874334f86a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 42803 } }
I20260929 18:46:16.348299 27990 heartbeater.cc:503] Master 127.7.157.126:33877 was elected leader, sending a full tablet report...
I20260929 18:46:16.348572 28007 ts_tablet_manager.cc:1428] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:46:16.348729 28009 ts_tablet_manager.cc:1428] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:46:16.349050 27859 heartbeater.cc:503] Master 127.7.157.126:33877 was elected leader, sending a full tablet report...
I20260929 18:46:16.348531 28008 raft_consensus.cc:348] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53401294ca44589baa9023466f9cf36" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35399 } } peers { permanent_uuid: "828d10c0e67447c28911b93b1a1da9ed" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40999 } } peers { permanent_uuid: "6977d98d07a64e6aa7117f874334f86a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 42803 } }
I20260929 18:46:16.349357 28008 raft_consensus.cc:374] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:16.349447 28008 raft_consensus.cc:729] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6977d98d07a64e6aa7117f874334f86a, State: Initialized, Role: FOLLOWER
I20260929 18:46:16.349895 28008 consensus_queue.cc:260] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53401294ca44589baa9023466f9cf36" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35399 } } peers { permanent_uuid: "828d10c0e67447c28911b93b1a1da9ed" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40999 } } peers { permanent_uuid: "6977d98d07a64e6aa7117f874334f86a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 42803 } }
I20260929 18:46:16.350498 27728 heartbeater.cc:503] Master 127.7.157.126:33877 was elected leader, sending a full tablet report...
I20260929 18:46:16.350829 28008 ts_tablet_manager.cc:1428] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260929 18:46:16.355460 27860 tablet.cc:2369] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:46:16.414474 27729 tablet.cc:2369] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:46:16.420306 28014 raft_consensus.cc:482] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:46:16.420528 28014 raft_consensus.cc:504] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53401294ca44589baa9023466f9cf36" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35399 } } peers { permanent_uuid: "828d10c0e67447c28911b93b1a1da9ed" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40999 } } peers { permanent_uuid: "6977d98d07a64e6aa7117f874334f86a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 42803 } }
I20260929 18:46:16.421656 28014 leader_election.cc:288] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 828d10c0e67447c28911b93b1a1da9ed (127.7.157.67:40999), 6977d98d07a64e6aa7117f874334f86a (127.7.157.65:42803)
I20260929 18:46:16.424474 28015 raft_consensus.cc:482] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:46:16.424665 28015 raft_consensus.cc:504] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53401294ca44589baa9023466f9cf36" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35399 } } peers { permanent_uuid: "828d10c0e67447c28911b93b1a1da9ed" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40999 } } peers { permanent_uuid: "6977d98d07a64e6aa7117f874334f86a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 42803 } }
I20260929 18:46:16.425279 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.421889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:40999 (local address 127.7.157.66:40087)
0929 18:46:16.422003 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:16.422018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:16.422113 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:16.422343 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:16.422346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:16.422364 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:16.422470 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.422477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.423206 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.423209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.423982 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.423991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.424888 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.424892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.424996 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.425020 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:46:16.425082 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:46:16.425157 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 18:46:16.425607 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.421995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:57183 (local address 127.7.157.65:42803)
0929 18:46:16.422123 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:16.422127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:16.422453 (+ 326us) server_negotiation.cc:407] Connection header received
0929 18:46:16.422572 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:16.422575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:16.422619 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:16.422680 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:16.422933 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.423464 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.424427 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.425005 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.425114 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.425154 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:16.425408 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 18:46:16.425498 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:46:16.425736 28004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.421995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.66:40087 (local address 127.7.157.67:40999)
0929 18:46:16.422117 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:16.422121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:16.422138 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:46:16.422200 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:16.422203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:16.422252 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:16.422331 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:16.422586 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.423096 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.424105 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.424646 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.424781 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.424829 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:16.425554 (+ 725us) server_negotiation.cc:299] Negotiation successful
0929 18:46:16.425622 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:46:16.425952 28016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.421889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:42803 (local address 127.7.157.66:57183)
0929 18:46:16.422364 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:16.422374 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:16.422475 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:16.422697 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:16.422700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:16.422715 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:16.422822 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.422827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.423565 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.423569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.424323 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.424331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.425151 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.425154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.425249 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.425274 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:46:16.425739 (+ 465us) client_negotiation.cc:240] Negotiation successful
0929 18:46:16.425815 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":161,"thread_start_us":112,"threads_started":1}
I20260929 18:46:16.426335 27944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a074f4195d94801847030a721e96de2" candidate_uuid: "b53401294ca44589baa9023466f9cf36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "828d10c0e67447c28911b93b1a1da9ed" is_pre_election: true
I20260929 18:46:16.426450 28015 leader_election.cc:288] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b53401294ca44589baa9023466f9cf36 (127.7.157.66:35399), 828d10c0e67447c28911b93b1a1da9ed (127.7.157.67:40999)
I20260929 18:46:16.426656 27944 raft_consensus.cc:2454] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b53401294ca44589baa9023466f9cf36 in term 0.
I20260929 18:46:16.427104 27682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a074f4195d94801847030a721e96de2" candidate_uuid: "b53401294ca44589baa9023466f9cf36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6977d98d07a64e6aa7117f874334f86a" is_pre_election: true
I20260929 18:46:16.427297 27682 raft_consensus.cc:2454] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b53401294ca44589baa9023466f9cf36 in term 0.
I20260929 18:46:16.427352 27746 leader_election.cc:302] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 828d10c0e67447c28911b93b1a1da9ed, b53401294ca44589baa9023466f9cf36; no voters:
I20260929 18:46:16.427644 28014 raft_consensus.cc:2781] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:46:16.427732 28014 raft_consensus.cc:482] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:46:16.428121 28014 raft_consensus.cc:3037] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:16.429353 28014 raft_consensus.cc:504] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53401294ca44589baa9023466f9cf36" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35399 } } peers { permanent_uuid: "828d10c0e67447c28911b93b1a1da9ed" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40999 } } peers { permanent_uuid: "6977d98d07a64e6aa7117f874334f86a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 42803 } }
I20260929 18:46:16.430001 27944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a074f4195d94801847030a721e96de2" candidate_uuid: "b53401294ca44589baa9023466f9cf36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "828d10c0e67447c28911b93b1a1da9ed"
I20260929 18:46:16.430127 27944 raft_consensus.cc:3037] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:16.430708 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.427138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:38803 (local address 127.7.157.66:35399)
0929 18:46:16.427242 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:16.427247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:16.427258 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:46:16.427292 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:16.427294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:16.427335 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:16.427403 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:16.427749 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.428317 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.429685 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.430165 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.430270 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.430309 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:16.430548 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 18:46:16.430611 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:46:16.430728 28017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.426290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:35399 (local address 127.7.157.65:38803)
0929 18:46:16.426861 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:16.426872 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:16.426960 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:16.427511 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:16.427515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:16.427532 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:16.427628 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.427634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.428458 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.428461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.429573 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.429583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.430360 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.430363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.430435 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.430455 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:46:16.430518 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:46:16.430584 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":21,"thread_start_us":75,"threads_started":1}
I20260929 18:46:16.431392 27944 raft_consensus.cc:2454] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b53401294ca44589baa9023466f9cf36 in term 1.
I20260929 18:46:16.431376 27682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a074f4195d94801847030a721e96de2" candidate_uuid: "b53401294ca44589baa9023466f9cf36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6977d98d07a64e6aa7117f874334f86a"
I20260929 18:46:16.431502 27682 raft_consensus.cc:3037] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:16.431573 28004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.427726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.157.65:36411 (local address 127.7.157.67:40999)
0929 18:46:16.427834 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:16.427839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:16.427852 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:46:16.427890 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:16.427892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:16.427931 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:16.427996 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:46:16.428258 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.428980 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.430406 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.430908 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.431003 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.431045 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:16.431409 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 18:46:16.431478 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:46:16.431720 27746 leader_election.cc:302] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 828d10c0e67447c28911b93b1a1da9ed, b53401294ca44589baa9023466f9cf36; no voters:
I20260929 18:46:16.431743 28014 leader_election.cc:288] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [CANDIDATE]: Term 1 election: Requested vote from peers 828d10c0e67447c28911b93b1a1da9ed (127.7.157.67:40999), 6977d98d07a64e6aa7117f874334f86a (127.7.157.65:42803)
I20260929 18:46:16.431705 27813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a074f4195d94801847030a721e96de2" candidate_uuid: "6977d98d07a64e6aa7117f874334f86a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53401294ca44589baa9023466f9cf36" is_pre_election: true
I20260929 18:46:16.431918 28018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.426248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.67:40999 (local address 127.7.157.65:36411)
0929 18:46:16.426525 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:16.426539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:16.426630 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:16.428021 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:16.428024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:16.428041 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:46:16.428156 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.428162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.429072 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.429076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.430276 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:16.430286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.431176 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.431179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:16.431270 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.431293 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:46:16.431713 (+ 420us) client_negotiation.cc:240] Negotiation successful
0929 18:46:16.431786 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260929 18:46:16.432035 27813 raft_consensus.cc:2379] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6977d98d07a64e6aa7117f874334f86a in current term 1: Already voted for candidate b53401294ca44589baa9023466f9cf36 in this term.
I20260929 18:46:16.432441 28014 raft_consensus.cc:2781] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:16.432542 27944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a074f4195d94801847030a721e96de2" candidate_uuid: "6977d98d07a64e6aa7117f874334f86a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "828d10c0e67447c28911b93b1a1da9ed" is_pre_election: true
I20260929 18:46:16.432648 27682 raft_consensus.cc:2454] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b53401294ca44589baa9023466f9cf36 in term 1.
I20260929 18:46:16.432664 28014 raft_consensus.cc:686] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [term 1 LEADER]: Becoming Leader. State: Replica: b53401294ca44589baa9023466f9cf36, State: Running, Role: LEADER
I20260929 18:46:16.432693 27944 raft_consensus.cc:2379] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6977d98d07a64e6aa7117f874334f86a in current term 1: Already voted for candidate b53401294ca44589baa9023466f9cf36 in this term.
I20260929 18:46:16.433104 27615 leader_election.cc:302] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6977d98d07a64e6aa7117f874334f86a; no voters: 828d10c0e67447c28911b93b1a1da9ed, b53401294ca44589baa9023466f9cf36
I20260929 18:46:16.433189 28014 consensus_queue.cc:237] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53401294ca44589baa9023466f9cf36" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35399 } } peers { permanent_uuid: "828d10c0e67447c28911b93b1a1da9ed" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40999 } } peers { permanent_uuid: "6977d98d07a64e6aa7117f874334f86a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 42803 } }
I20260929 18:46:16.433362 28015 raft_consensus.cc:2726] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 18:46:16.436329 27538 catalog_manager.cc:5495] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 reported cstate change: term changed from 0 to 1, leader changed from <none> to b53401294ca44589baa9023466f9cf36 (127.7.157.66). New cstate: current_term: 1 leader_uuid: "b53401294ca44589baa9023466f9cf36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53401294ca44589baa9023466f9cf36" member_type: VOTER last_known_addr { host: "127.7.157.66" port: 35399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "828d10c0e67447c28911b93b1a1da9ed" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6977d98d07a64e6aa7117f874334f86a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 42803 } health_report { overall_health: UNKNOWN } } }
I20260929 18:46:16.446974 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.443975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54980 (local address 127.7.157.66:35399)
0929 18:46:16.444085 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:16.444089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:16.444101 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:46:16.444216 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:16.444218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:16.444269 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:16.444350 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:46:16.444540 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.445025 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.445795 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:16.446002 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:16.446070 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:16.446278 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:46:16.446689 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:46:16.446772 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:16.446798 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 18:46:16.446841 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:46:16.452451 27944 raft_consensus.cc:1261] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed [term 1 FOLLOWER]: Refusing update from remote peer b53401294ca44589baa9023466f9cf36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:46:16.452485 27682 raft_consensus.cc:1261] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a [term 1 FOLLOWER]: Refusing update from remote peer b53401294ca44589baa9023466f9cf36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:46:16.453068 28019 consensus_queue.cc:1035] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "828d10c0e67447c28911b93b1a1da9ed" member_type: VOTER last_known_addr { host: "127.7.157.67" port: 40999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:46:16.453226 28014 consensus_queue.cc:1035] T 8a074f4195d94801847030a721e96de2 P b53401294ca44589baa9023466f9cf36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977d98d07a64e6aa7117f874334f86a" member_type: VOTER last_known_addr { host: "127.7.157.65" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:46:16.460249 28024 mvcc.cc:205] Tried to move back new op lower bound from 7334738233142562816 to 7334738233072889856. Current Snapshot: MvccSnapshot[applied={T|T < 7334738233142562816}]
I20260929 18:46:16.461633 28023 mvcc.cc:205] Tried to move back new op lower bound from 7334738233142562816 to 7334738233072889856. Current Snapshot: MvccSnapshot[applied={T|T < 7334738233142562816}]
I20260929 18:46:16.462343 28022 mvcc.cc:205] Tried to move back new op lower bound from 7334738233142562816 to 7334738233072889856. Current Snapshot: MvccSnapshot[applied={T|T < 7334738233142562816}]
I20260929 18:46:16.468590 27539 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:46:16.468729 27539 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:46:16.470925 27539 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94b27b2f08644048a358fd7f83c0e8ef: Sending DeleteTablet for 3 replicas of tablet 8a074f4195d94801847030a721e96de2
I20260929 18:46:16.471715 27924 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a074f4195d94801847030a721e96de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:16 UTC) from {username='slave'} at 127.0.0.1:42994
I20260929 18:46:16.471758 27662 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a074f4195d94801847030a721e96de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:16 UTC) from {username='slave'} at 127.0.0.1:59702
I20260929 18:46:16.472229 28032 tablet_replica.cc:330] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a: stopping tablet replica
I20260929 18:46:16.472352 27793 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a074f4195d94801847030a721e96de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:46:16 UTC) from {username='slave'} at 127.0.0.1:54964
I20260929 18:46:16.472676 28032 raft_consensus.cc:2229] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:46:16.472806 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 27600
I20260929 18:46:16.473029 28032 raft_consensus.cc:2258] T 8a074f4195d94801847030a721e96de2 P 6977d98d07a64e6aa7117f874334f86a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:46:16.473362 28033 tablet_replica.cc:330] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed: stopping tablet replica
I20260929 18:46:16.473662 28033 raft_consensus.cc:2229] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:46:16.473932 28033 raft_consensus.cc:2258] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:46:16.475500 28033 ts_tablet_manager.cc:1905] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:46:16.478262 28033 ts_tablet_manager.cc:1918] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:46:16.478358 28033 log.cc:1196] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed: Deleting WAL directory at /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707328618849-7797-0/minicluster-data/ts-2/wal/wals/8a074f4195d94801847030a721e96de2
I20260929 18:46:16.478657 28033 ts_tablet_manager.cc:1939] T 8a074f4195d94801847030a721e96de2 P 828d10c0e67447c28911b93b1a1da9ed: Deleting consensus metadata
W20260929 18:46:16.479724 27527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.157.65:42803 (error 108)
I20260929 18:46:16.479943 27524 catalog_manager.cc:4841] TS 828d10c0e67447c28911b93b1a1da9ed (127.7.157.67:40999): tablet 8a074f4195d94801847030a721e96de2 (table test-table [id=1277e013aeaf44fe9e7d51d5a9e5c4f9]) successfully deleted
I20260929 18:46:16.480255 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 27731
I20260929 18:46:16.481762 28002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.481038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.65:42803 (local address 127.0.0.1:59710)
0929 18:46:16.481621 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:16.481688 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.65:42803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501}
W20260929 18:46:16.481910 27527 catalog_manager.cc:4584] TS 6977d98d07a64e6aa7117f874334f86a (127.7.157.65:42803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a074f4195d94801847030a721e96de2: Network error: Client connection negotiation failed: client connection to 127.7.157.65:42803: connect: Connection refused (error 111)
I20260929 18:46:16.486048 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 27862
I20260929 18:46:16.486756 28002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:16.486200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.66:35399 (local address 127.0.0.1:54990)
0929 18:46:16.486625 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:16.486677 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.66:35399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353}
I20260929 18:46:16.490412 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 27509
2026-09-29T18:46:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1032 ms)
[----------] 6 tests from EncryptionPolicyTest (10126 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:16 dist-test-slave-smpn krb5kdc[28038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:16 dist-test-slave-smpn krb5kdc[28038](info): set up 2 sockets
Sep 29 18:46:16 dist-test-slave-smpn krb5kdc[28038](info): commencing operation
krb5kdc: starting...
W20260929 18:46:17.450911 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:17 dist-test-slave-smpn krb5kdc[28038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:46:17.569576 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33415
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:17.677042 28051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:17.677309 28051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:17.677354 28051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:17.681398 28051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:17.681470 28051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:17.681489 28051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:17.681507 28051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:17.681535 28051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:17.686893 28051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33415
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:17.688007 28051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:17.689008 28051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:17.697384 28056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:17.697510 28051 server_base.cc:1034] running on GCE node
W20260929 18:46:17.697372 28059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:17.697369 28057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:17.698194 28051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:17.699436 28051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:17.700585 28051 hybrid_clock.cc:648] HybridClock initialized: now 1790707577700559 us; error 39 us; skew 500 ppm
Sep 29 18:46:17 dist-test-slave-smpn krb5kdc[28038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:17.703260 28051 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:46:17.704177 28051 webserver.cc:460] Webserver started at http://127.0.0.1:37413/ using document root <none> and password file <none>
I20260929 18:46:17.704718 28051 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:17.704780 28051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:17.705008 28051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:17.707094 28051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "7709611a9feb4756b07cf82a32302fc7"
format_stamp: "Formatted at 2026-09-29 18:46:17 on dist-test-slave-smpn"
server_key: "68b92b2327a4d70fe7692b2b2ce42356"
server_key_iv: "622796dccba8e7da68c78a17b0251db1"
server_key_version: "encryptionkey@0"
I20260929 18:46:17.707688 28051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "7709611a9feb4756b07cf82a32302fc7"
format_stamp: "Formatted at 2026-09-29 18:46:17 on dist-test-slave-smpn"
server_key: "68b92b2327a4d70fe7692b2b2ce42356"
server_key_iv: "622796dccba8e7da68c78a17b0251db1"
server_key_version: "encryptionkey@0"
I20260929 18:46:17.711478 28051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 18:46:17.713923 28066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:17.715232 28051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:17.715351 28051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "7709611a9feb4756b07cf82a32302fc7"
format_stamp: "Formatted at 2026-09-29 18:46:17 on dist-test-slave-smpn"
server_key: "68b92b2327a4d70fe7692b2b2ce42356"
server_key_iv: "622796dccba8e7da68c78a17b0251db1"
server_key_version: "encryptionkey@0"
I20260929 18:46:17.715461 28051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:17.732456 28051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:17.735320 28051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:17.735498 28051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:17.742239 28051 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33415
I20260929 18:46:17.742252 28118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33415 every 8 connection(s)
I20260929 18:46:17.743230 28051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:17.745796 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 28051
I20260929 18:46:17.745935 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:17.746013 28119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:17.746246 7797 external_mini_cluster.cc:1428] Setting key 429301090d8efd25cd43010106ce097c
I20260929 18:46:17.752996 28119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7: Bootstrap starting.
Sep 29 18:46:17 dist-test-slave-smpn krb5kdc[28038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:46:17.755693 28119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:17.756531 28119 log.cc:824] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:17.758627 28119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7: No bootstrap required, opened a new log
I20260929 18:46:17.760731 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:17.747637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40217 (local address 127.0.0.1:33415)
0929 18:46:17.748057 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:17.748072 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:17.748114 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:46:17.749001 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:17.749027 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:17.749440 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:17.749726 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:17.750567 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:17.751626 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:17.752740 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:17.753249 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:17.753364 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:17.756118 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:17.756124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:17.756137 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:17.756165 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:17.757853 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:17.758369 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:17.758374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:17.758380 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:17.758497 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:17.758825 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:17.758828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:17.758829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:17.759275 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:17.759449 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:17.759731 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 18:46:17.759962 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":117,"threads_started":1}
I20260929 18:46:17.762305 28119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709611a9feb4756b07cf82a32302fc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33415 } }
I20260929 18:46:17.762542 28119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:17.762595 28119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7709611a9feb4756b07cf82a32302fc7, State: Initialized, Role: FOLLOWER
I20260929 18:46:17.763038 28119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709611a9feb4756b07cf82a32302fc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33415 } }
I20260929 18:46:17.763204 28119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:17.763273 28119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:17.763371 28119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:17.763674 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:17.764493 28119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709611a9feb4756b07cf82a32302fc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33415 } }
I20260929 18:46:17.764863 28119 leader_election.cc:302] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7709611a9feb4756b07cf82a32302fc7; no voters:
I20260929 18:46:17.765187 28119 leader_election.cc:288] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:17.765309 28128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:17.765549 28128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 7709611a9feb4756b07cf82a32302fc7, State: Running, Role: LEADER
I20260929 18:46:17.765851 28128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709611a9feb4756b07cf82a32302fc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33415 } }
I20260929 18:46:17.766506 28119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:17.767127 28129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7709611a9feb4756b07cf82a32302fc7. Latest consensus state: current_term: 1 leader_uuid: "7709611a9feb4756b07cf82a32302fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709611a9feb4756b07cf82a32302fc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33415 } } }
I20260929 18:46:17.767254 28129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:17.767549 28130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7709611a9feb4756b07cf82a32302fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709611a9feb4756b07cf82a32302fc7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33415 } } }
I20260929 18:46:17.767655 28130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7 [sys.catalog]: This master's current role is: LEADER
W20260929 18:46:17.771847 28144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:46:17.771932 28144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:46:17.771992 28137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:17.772692 28137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:17.774880 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:17.766118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52729 (local address 127.0.0.1:33415)
0929 18:46:17.766216 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:17.766220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:17.766235 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:46:17.766341 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:17.766344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:17.766389 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:17.766446 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:17.767590 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:17.768303 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:17.769392 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:17.769644 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:17.769724 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:17.771201 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:17.771205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:17.771208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:17.771231 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:17.772833 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:17.773347 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:17.773351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:17.773353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:17.773449 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:17.773841 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:17.773845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:17.773847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:17.774056 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:17.774173 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:17.774632 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 18:46:17.774732 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:46:17.778313 28137 catalog_manager.cc:1347] Generated new cluster ID: d04372231abe43489bce45f14ae8fdc0
I20260929 18:46:17.778378 28137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:17.789791 28137 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:17.790707 28137 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:17.799793 28137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7709611a9feb4756b07cf82a32302fc7: Generated new TSK 0
I20260929 18:46:17.800637 28137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:17.809561 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 28051
W20260929 18:46:17.814383 28036 connection.cc:535] client connection to 127.0.0.1:33415 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33415): Connection reset by peer (error 104)
2026-09-29T18:46:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:17 dist-test-slave-smpn krb5kdc[28150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:17 dist-test-slave-smpn krb5kdc[28150](info): set up 2 sockets
Sep 29 18:46:17 dist-test-slave-smpn krb5kdc[28150](info): commencing operation
krb5kdc: starting...
W20260929 18:46:18.759119 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:18 dist-test-slave-smpn krb5kdc[28150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:46:18.874517 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:18.981536 28163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:18.981820 28163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:18.981861 28163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:18.985864 28163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:18.985942 28163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:18.985961 28163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:18.985976 28163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:18.985991 28163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:18.991484 28163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:18.992565 28163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:18.993558 28163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:19.002137 28171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:19.002137 28169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:19.002234 28168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:19.002449 28163 server_base.cc:1034] running on GCE node
I20260929 18:46:19.003116 28163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:19.004338 28163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:19.005537 28163 hybrid_clock.cc:648] HybridClock initialized: now 1790707579005517 us; error 30 us; skew 500 ppm
Sep 29 18:46:19 dist-test-slave-smpn krb5kdc[28150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:19.008390 28163 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:46:19.009268 28163 webserver.cc:460] Webserver started at http://127.0.0.1:33877/ using document root <none> and password file <none>
I20260929 18:46:19.009797 28163 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:19.009908 28163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:19.010115 28163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:19.012202 28163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "23db3adf79364ad8ae273cf37bb903f3"
format_stamp: "Formatted at 2026-09-29 18:46:19 on dist-test-slave-smpn"
server_key: "89dc0e39ba1505dbaa6d89941251698d"
server_key_iv: "2895537fac8948611d249b7e7f6e6b71"
server_key_version: "encryptionkey@0"
I20260929 18:46:19.012722 28163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "23db3adf79364ad8ae273cf37bb903f3"
format_stamp: "Formatted at 2026-09-29 18:46:19 on dist-test-slave-smpn"
server_key: "89dc0e39ba1505dbaa6d89941251698d"
server_key_iv: "2895537fac8948611d249b7e7f6e6b71"
server_key_version: "encryptionkey@0"
I20260929 18:46:19.016387 28163 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:46:19.018810 28178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:19.020102 28163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:19.020221 28163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "23db3adf79364ad8ae273cf37bb903f3"
format_stamp: "Formatted at 2026-09-29 18:46:19 on dist-test-slave-smpn"
server_key: "89dc0e39ba1505dbaa6d89941251698d"
server_key_iv: "2895537fac8948611d249b7e7f6e6b71"
server_key_version: "encryptionkey@0"
I20260929 18:46:19.020325 28163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:19.034904 28163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:19.037575 28163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:19.037748 28163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:19.044541 28163 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39825
I20260929 18:46:19.044551 28230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39825 every 8 connection(s)
I20260929 18:46:19.045509 28163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:19.048311 28231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:19.050058 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 28163
I20260929 18:46:19.050191 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:19.050455 7797 external_mini_cluster.cc:1428] Setting key a3f62413903f2ff18047a3be387b43a7
I20260929 18:46:19.055054 28231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3: Bootstrap starting.
I20260929 18:46:19.057682 28231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:19.058498 28231 log.cc:824] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:19.060107 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:19.051886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47789 (local address 127.0.0.1:39825)
0929 18:46:19.052293 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:19.052312 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:19.052356 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:46:19.053148 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:19.053174 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:19.053623 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:19.053999 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:19.054399 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:19.055814 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:19.056660 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:19.057336 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:19.057491 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:19.058231 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:19.058237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:19.058253 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:19.058749 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:19.058813 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:19.058846 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:19.058981 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:19.059209 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 18:46:19.059343 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":96,"threads_started":1}
I20260929 18:46:19.060562 28231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3: No bootstrap required, opened a new log
I20260929 18:46:19.063000 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:19.064839 28231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23db3adf79364ad8ae273cf37bb903f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39825 } }
I20260929 18:46:19.065105 28231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:19.065164 28231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23db3adf79364ad8ae273cf37bb903f3, State: Initialized, Role: FOLLOWER
I20260929 18:46:19.065606 28231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23db3adf79364ad8ae273cf37bb903f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39825 } }
I20260929 18:46:19.065745 28231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:19.065807 28231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:19.065905 28231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:19.066999 28231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23db3adf79364ad8ae273cf37bb903f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39825 } }
I20260929 18:46:19.067342 28231 leader_election.cc:302] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23db3adf79364ad8ae273cf37bb903f3; no voters:
I20260929 18:46:19.067636 28231 leader_election.cc:288] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:19.067742 28241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:19.067919 28241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [term 1 LEADER]: Becoming Leader. State: Replica: 23db3adf79364ad8ae273cf37bb903f3, State: Running, Role: LEADER
I20260929 18:46:19.068217 28241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23db3adf79364ad8ae273cf37bb903f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39825 } }
I20260929 18:46:19.068975 28231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:19.069487 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:19.065248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56371 (local address 127.0.0.1:39825)
0929 18:46:19.065355 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:19.065360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:19.065549 (+ 189us) server_negotiation.cc:407] Connection header received
0929 18:46:19.065659 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:19.065662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:19.065702 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:19.065763 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:19.066040 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:19.066762 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:19.067692 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:19.068020 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:19.068129 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:19.069019 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:19.069022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:19.069025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:19.069038 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:19.069065 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:19.069077 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:19.069154 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:19.069324 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 18:46:19.069358 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:46:19.069938 28242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23db3adf79364ad8ae273cf37bb903f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23db3adf79364ad8ae273cf37bb903f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39825 } } }
I20260929 18:46:19.070066 28242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:19.070351 28243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23db3adf79364ad8ae273cf37bb903f3. Latest consensus state: current_term: 1 leader_uuid: "23db3adf79364ad8ae273cf37bb903f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23db3adf79364ad8ae273cf37bb903f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39825 } } }
I20260929 18:46:19.070447 28243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:19.070556 28250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:19.073777 28250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:19.079039 28250 catalog_manager.cc:1347] Generated new cluster ID: b63389ca9d364cd488188c65ca40ede2
I20260929 18:46:19.079129 28250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:19.090893 28250 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:19.091835 28250 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:19.100507 28250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23db3adf79364ad8ae273cf37bb903f3: Generated new TSK 0
I20260929 18:46:19.101190 28250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:19.108981 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 28163
W20260929 18:46:19.113824 28148 connection.cc:535] client connection to 127.0.0.1:39825 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39825): Connection reset by peer (error 104)
2026-09-29T18:46:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:19 dist-test-slave-smpn krb5kdc[28262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:19 dist-test-slave-smpn krb5kdc[28262](info): set up 2 sockets
Sep 29 18:46:19 dist-test-slave-smpn krb5kdc[28262](info): commencing operation
krb5kdc: starting...
W20260929 18:46:20.058951 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:20 dist-test-slave-smpn krb5kdc[28262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:46:20.174356 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:20.283721 28275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:20.284025 28275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:20.284088 28275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:20.288215 28275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:20.288293 28275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:20.288321 28275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:20.288341 28275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:20.288368 28275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:20.295168 28275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:20.296805 28275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:20.298200 28275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:20.307315 28281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:20.307315 28280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:20.307415 28275 server_base.cc:1034] running on GCE node
W20260929 18:46:20.307350 28283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:20.308128 28275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:20.309235 28275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:20.310381 28275 hybrid_clock.cc:648] HybridClock initialized: now 1790707580310360 us; error 40 us; skew 500 ppm
Sep 29 18:46:20 dist-test-slave-smpn krb5kdc[28262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:20.313146 28275 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:46:20.313982 28275 webserver.cc:460] Webserver started at http://127.0.0.1:34903/ using document root <none> and password file <none>
I20260929 18:46:20.314497 28275 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:20.314560 28275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:20.314738 28275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:20.316793 28275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "e475eb2254c2449fbbce2441a28f8bc5"
format_stamp: "Formatted at 2026-09-29 18:46:20 on dist-test-slave-smpn"
server_key: "de594435f32fe0536f871c25f12685c9"
server_key_iv: "ddf7bf93294b2f8e143dda3c4b53a242"
server_key_version: "encryptionkey@0"
I20260929 18:46:20.317328 28275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "e475eb2254c2449fbbce2441a28f8bc5"
format_stamp: "Formatted at 2026-09-29 18:46:20 on dist-test-slave-smpn"
server_key: "de594435f32fe0536f871c25f12685c9"
server_key_iv: "ddf7bf93294b2f8e143dda3c4b53a242"
server_key_version: "encryptionkey@0"
I20260929 18:46:20.320968 28275 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:46:20.323339 28290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:20.324646 28275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:46:20.324769 28275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "e475eb2254c2449fbbce2441a28f8bc5"
format_stamp: "Formatted at 2026-09-29 18:46:20 on dist-test-slave-smpn"
server_key: "de594435f32fe0536f871c25f12685c9"
server_key_iv: "ddf7bf93294b2f8e143dda3c4b53a242"
server_key_version: "encryptionkey@0"
I20260929 18:46:20.324872 28275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:20.329365 28275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:20.332371 28275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:20.332531 28275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:20.338810 28275 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36053
I20260929 18:46:20.338828 28342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36053 every 8 connection(s)
I20260929 18:46:20.339772 28275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:20.340055 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 28275
I20260929 18:46:20.340170 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:20.340487 7797 external_mini_cluster.cc:1428] Setting key f4736e1fd905ca7945ad360fdb0cafe3
I20260929 18:46:20.342783 28343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:20.345500 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:20.341857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53893 (local address 127.0.0.1:36053)
0929 18:46:20.342285 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:20.342303 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:20.342339 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:46:20.343000 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:20.343018 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:20.343349 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:20.343654 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:20.344399 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:20.344404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:20.344416 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:20.344426 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:20.344477 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:20.344495 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:20.344619 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:20.344685 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 18:46:20.344839 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":128,"threads_started":1}
I20260929 18:46:20.348064 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:20.348621 28343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5: Bootstrap starting.
I20260929 18:46:20.351220 28343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:20.351854 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:20.350272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48831 (local address 127.0.0.1:36053)
0929 18:46:20.350383 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:20.350388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:20.350629 (+ 241us) server_negotiation.cc:407] Connection header received
0929 18:46:20.350736 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:20.350739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:20.350769 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:20.350830 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:20.351500 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:20.351502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:20.351505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:20.351514 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:20.351539 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:20.351549 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:20.351618 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:20.351709 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 18:46:20.351740 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:46:20.352154 28343 log.cc:824] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:20.353997 28343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5: No bootstrap required, opened a new log
I20260929 18:46:20.357409 28343 raft_consensus.cc:348] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e475eb2254c2449fbbce2441a28f8bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36053 } }
I20260929 18:46:20.357630 28343 raft_consensus.cc:374] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:20.357684 28343 raft_consensus.cc:729] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e475eb2254c2449fbbce2441a28f8bc5, State: Initialized, Role: FOLLOWER
I20260929 18:46:20.358071 28343 consensus_queue.cc:260] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e475eb2254c2449fbbce2441a28f8bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36053 } }
I20260929 18:46:20.358196 28343 raft_consensus.cc:388] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:20.358258 28343 raft_consensus.cc:482] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:20.358353 28343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:20.359226 28343 raft_consensus.cc:504] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e475eb2254c2449fbbce2441a28f8bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36053 } }
I20260929 18:46:20.359542 28343 leader_election.cc:302] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e475eb2254c2449fbbce2441a28f8bc5; no voters:
I20260929 18:46:20.359819 28343 leader_election.cc:288] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:20.359915 28353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:20.360148 28353 raft_consensus.cc:686] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [term 1 LEADER]: Becoming Leader. State: Replica: e475eb2254c2449fbbce2441a28f8bc5, State: Running, Role: LEADER
I20260929 18:46:20.360448 28353 consensus_queue.cc:237] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e475eb2254c2449fbbce2441a28f8bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36053 } }
I20260929 18:46:20.360976 28343 sys_catalog.cc:564] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:20.361941 28354 sys_catalog.cc:455] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e475eb2254c2449fbbce2441a28f8bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e475eb2254c2449fbbce2441a28f8bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36053 } } }
I20260929 18:46:20.362068 28354 sys_catalog.cc:458] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:20.362339 28355 sys_catalog.cc:455] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e475eb2254c2449fbbce2441a28f8bc5. Latest consensus state: current_term: 1 leader_uuid: "e475eb2254c2449fbbce2441a28f8bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e475eb2254c2449fbbce2441a28f8bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36053 } } }
I20260929 18:46:20.362432 28355 sys_catalog.cc:458] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:20.362574 28362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:20.366103 28362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:20.372146 28362 catalog_manager.cc:1347] Generated new cluster ID: 76c9d61034494a34aa6c40466c1df094
I20260929 18:46:20.372210 28362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:20.387905 28362 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:20.388201 28362 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:20.399016 28362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e475eb2254c2449fbbce2441a28f8bc5: Generated new TSK 0
I20260929 18:46:20.399708 28362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:20.406430 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 28275
2026-09-29T18:46:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:20 dist-test-slave-smpn krb5kdc[28374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:20 dist-test-slave-smpn krb5kdc[28374](info): set up 2 sockets
Sep 29 18:46:20 dist-test-slave-smpn krb5kdc[28374](info): commencing operation
krb5kdc: starting...
W20260929 18:46:21.362993 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:21 dist-test-slave-smpn krb5kdc[28374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:46:21.479099 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37737
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:21.585685 28387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:21.585990 28387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:21.586052 28387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:21.590134 28387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:21.590207 28387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:21.590234 28387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:21.590255 28387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:21.590276 28387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:21.595640 28387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:21.596750 28387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:21.597771 28387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:21.606340 28392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:21.606340 28393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:21.606410 28387 server_base.cc:1034] running on GCE node
W20260929 18:46:21.606647 28395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:21.607234 28387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:21.608333 28387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:21.609493 28387 hybrid_clock.cc:648] HybridClock initialized: now 1790707581609467 us; error 44 us; skew 500 ppm
Sep 29 18:46:21 dist-test-slave-smpn krb5kdc[28374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:21.618031 28387 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:46:21.619042 28387 webserver.cc:460] Webserver started at http://127.0.0.1:35237/ using document root <none> and password file <none>
I20260929 18:46:21.619664 28387 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:21.619737 28387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:21.619948 28387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:21.622057 28387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "c66a679c40b544f7b7a6715b95852632"
format_stamp: "Formatted at 2026-09-29 18:46:21 on dist-test-slave-smpn"
server_key: "0d3178e4652b5fb43609440f528aa470"
server_key_iv: "38d9110cfd5408d2e5a36582e85c4bf7"
server_key_version: "encryptionkey@0"
I20260929 18:46:21.622609 28387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "c66a679c40b544f7b7a6715b95852632"
format_stamp: "Formatted at 2026-09-29 18:46:21 on dist-test-slave-smpn"
server_key: "0d3178e4652b5fb43609440f528aa470"
server_key_iv: "38d9110cfd5408d2e5a36582e85c4bf7"
server_key_version: "encryptionkey@0"
I20260929 18:46:21.626526 28387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:46:21.629045 28402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:21.630347 28387 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:46:21.630491 28387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "c66a679c40b544f7b7a6715b95852632"
format_stamp: "Formatted at 2026-09-29 18:46:21 on dist-test-slave-smpn"
server_key: "0d3178e4652b5fb43609440f528aa470"
server_key_iv: "38d9110cfd5408d2e5a36582e85c4bf7"
server_key_version: "encryptionkey@0"
I20260929 18:46:21.630605 28387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:21.644649 28387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:21.647519 28387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:21.647696 28387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:21.654220 28387 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37737
I20260929 18:46:21.654256 28454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37737 every 8 connection(s)
I20260929 18:46:21.655200 28387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:21.657992 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:21.663853 28455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632: Bootstrap starting.
I20260929 18:46:21.664834 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 28387
I20260929 18:46:21.664973 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:21.665217 7797 external_mini_cluster.cc:1428] Setting key 271b52ce4f01759e1c236e2578a08e5a
I20260929 18:46:21.666702 28455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:21.667644 28455 log.cc:824] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:21.669912 28455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632: No bootstrap required, opened a new log
I20260929 18:46:21.674067 28455 raft_consensus.cc:348] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66a679c40b544f7b7a6715b95852632" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37737 } }
I20260929 18:46:21.674373 28455 raft_consensus.cc:374] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:21.674444 28455 raft_consensus.cc:729] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66a679c40b544f7b7a6715b95852632, State: Initialized, Role: FOLLOWER
Sep 29 18:46:21 dist-test-slave-smpn krb5kdc[28374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:46:21.674903 28455 consensus_queue.cc:260] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66a679c40b544f7b7a6715b95852632" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37737 } }
I20260929 18:46:21.675060 28455 raft_consensus.cc:388] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:21.675153 28455 raft_consensus.cc:482] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:21.675253 28455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:21.676462 28455 raft_consensus.cc:504] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66a679c40b544f7b7a6715b95852632" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37737 } }
I20260929 18:46:21.676849 28455 leader_election.cc:302] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c66a679c40b544f7b7a6715b95852632; no voters:
I20260929 18:46:21.677214 28455 leader_election.cc:288] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:21.677322 28460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:21.677542 28460 raft_consensus.cc:686] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [term 1 LEADER]: Becoming Leader. State: Replica: c66a679c40b544f7b7a6715b95852632, State: Running, Role: LEADER
I20260929 18:46:21.677879 28460 consensus_queue.cc:237] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66a679c40b544f7b7a6715b95852632" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37737 } }
I20260929 18:46:21.678610 28455 sys_catalog.cc:564] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:21.679827 28462 sys_catalog.cc:455] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c66a679c40b544f7b7a6715b95852632. Latest consensus state: current_term: 1 leader_uuid: "c66a679c40b544f7b7a6715b95852632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66a679c40b544f7b7a6715b95852632" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37737 } } }
I20260929 18:46:21.679828 28461 sys_catalog.cc:455] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c66a679c40b544f7b7a6715b95852632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66a679c40b544f7b7a6715b95852632" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37737 } } }
I20260929 18:46:21.679989 28462 sys_catalog.cc:458] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:21.679989 28461 sys_catalog.cc:458] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:21.680856 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:21.666335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36631 (local address 127.0.0.1:37737)
0929 18:46:21.666842 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:21.666866 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:21.666928 (+ 62us) server_negotiation.cc:407] Connection header received
0929 18:46:21.667845 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:21.667872 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:21.668306 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:21.668688 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:21.669342 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:21.670800 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:21.671686 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:21.672415 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:21.672587 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:21.675118 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:21.675126 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:21.675146 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:21.675182 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:21.677099 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:21.677652 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:21.677657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:21.677664 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:21.677807 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:21.678209 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:21.678212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:21.678214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:21.678748 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:21.678953 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:21.679315 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 18:46:21.679669 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":140,"threads_started":1}
W20260929 18:46:21.683805 28475 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:46:21.683910 28475 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:46:21.683974 28469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:21.684530 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:21.684856 28469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:21.691145 28469 catalog_manager.cc:1347] Generated new cluster ID: 71529ff4ebb84452bb15c570b32a6bbe
I20260929 18:46:21.691220 28469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:21.695398 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:21.686826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59701 (local address 127.0.0.1:37737)
0929 18:46:21.686934 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:21.686939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:21.687216 (+ 277us) server_negotiation.cc:407] Connection header received
0929 18:46:21.687346 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:21.687349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:21.687411 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:21.687490 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:21.688291 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:21.689123 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:21.690064 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:21.690312 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:21.690401 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:21.692049 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:21.692053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:21.692056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:21.692083 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:21.693726 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:21.694162 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:21.694165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:21.694167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:21.694260 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:21.694508 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:21.694511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:21.694513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:21.694734 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:21.694858 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:21.695098 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 18:46:21.695221 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 18:46:21.698347 28469 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:21.699347 28469 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:21.706054 28469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c66a679c40b544f7b7a6715b95852632: Generated new TSK 0
I20260929 18:46:21.706862 28469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:21.714671 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 28387
2026-09-29T18:46:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:21 dist-test-slave-smpn krb5kdc[28486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:21 dist-test-slave-smpn krb5kdc[28486](info): set up 2 sockets
Sep 29 18:46:21 dist-test-slave-smpn krb5kdc[28486](info): commencing operation
krb5kdc: starting...
W20260929 18:46:22.663060 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:22 dist-test-slave-smpn krb5kdc[28486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:46:22.778455 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38523
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:22.886439 28499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:22.886698 28499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:22.886754 28499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:22.890712 28499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:22.890770 28499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:22.890789 28499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:22.890803 28499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:22.890820 28499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:22.896009 28499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:22.896987 28499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:22.897943 28499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:22.906394 28507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:22.906422 28504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:22.906425 28505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:22.906481 28499 server_base.cc:1034] running on GCE node
I20260929 18:46:22.907178 28499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:22.908308 28499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:22.909473 28499 hybrid_clock.cc:648] HybridClock initialized: now 1790707582909451 us; error 34 us; skew 500 ppm
Sep 29 18:46:22 dist-test-slave-smpn krb5kdc[28486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:22.912202 28499 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:46:22.913108 28499 webserver.cc:460] Webserver started at http://127.0.0.1:39717/ using document root <none> and password file <none>
I20260929 18:46:22.913682 28499 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:22.913753 28499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:22.913957 28499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:22.916152 28499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "39e48ed968c749caac18330262f12ef5"
format_stamp: "Formatted at 2026-09-29 18:46:22 on dist-test-slave-smpn"
server_key: "874af7b9ee7638afbc0bd09cd9972f50"
server_key_iv: "b5c4189a0013964d9e157cb8d8d29ef4"
server_key_version: "encryptionkey@0"
I20260929 18:46:22.916697 28499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "39e48ed968c749caac18330262f12ef5"
format_stamp: "Formatted at 2026-09-29 18:46:22 on dist-test-slave-smpn"
server_key: "874af7b9ee7638afbc0bd09cd9972f50"
server_key_iv: "b5c4189a0013964d9e157cb8d8d29ef4"
server_key_version: "encryptionkey@0"
I20260929 18:46:22.920472 28499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:46:22.922794 28514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:22.924103 28499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:22.924219 28499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "39e48ed968c749caac18330262f12ef5"
format_stamp: "Formatted at 2026-09-29 18:46:22 on dist-test-slave-smpn"
server_key: "874af7b9ee7638afbc0bd09cd9972f50"
server_key_iv: "b5c4189a0013964d9e157cb8d8d29ef4"
server_key_version: "encryptionkey@0"
I20260929 18:46:22.924315 28499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:22.946324 28499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:22.949118 28499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:22.949302 28499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:22.956128 28499 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38523
I20260929 18:46:22.956154 28566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38523 every 8 connection(s)
I20260929 18:46:22.957077 28499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:22.959753 28567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:22.964123 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 28499
I20260929 18:46:22.964243 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:22.964504 7797 external_mini_cluster.cc:1428] Setting key ad60dd93c45c12859621fab6f3bd057a
I20260929 18:46:22.965494 28567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5: Bootstrap starting.
I20260929 18:46:22.967902 28567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:22.968618 28567 log.cc:824] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:22.970429 28567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5: No bootstrap required, opened a new log
I20260929 18:46:22.973088 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:22.965936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37339 (local address 127.0.0.1:38523)
0929 18:46:22.966378 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:22.966393 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:22.966429 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:46:22.967101 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:22.967125 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:22.967475 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:22.967787 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:22.968196 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:22.969200 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:22.970059 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:22.970525 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:22.970678 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:22.971476 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:22.971482 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:22.971497 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:22.971877 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:22.971930 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:22.971952 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:22.972034 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:22.972174 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 18:46:22.972378 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":134,"threads_started":1}
I20260929 18:46:22.974033 28567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e48ed968c749caac18330262f12ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38523 } }
I20260929 18:46:22.974257 28567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:22.974311 28567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e48ed968c749caac18330262f12ef5, State: Initialized, Role: FOLLOWER
I20260929 18:46:22.974718 28567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e48ed968c749caac18330262f12ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38523 } }
I20260929 18:46:22.974857 28567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:22.974956 28567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:22.975037 28567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:22.975841 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:22.976024 28567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e48ed968c749caac18330262f12ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38523 } }
I20260929 18:46:22.976341 28567 leader_election.cc:302] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39e48ed968c749caac18330262f12ef5; no voters:
I20260929 18:46:22.976629 28567 leader_election.cc:288] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:22.976727 28572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:22.976945 28572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [term 1 LEADER]: Becoming Leader. State: Replica: 39e48ed968c749caac18330262f12ef5, State: Running, Role: LEADER
I20260929 18:46:22.977330 28572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e48ed968c749caac18330262f12ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38523 } }
I20260929 18:46:22.977911 28567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:22.978549 28575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39e48ed968c749caac18330262f12ef5. Latest consensus state: current_term: 1 leader_uuid: "39e48ed968c749caac18330262f12ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e48ed968c749caac18330262f12ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38523 } } }
I20260929 18:46:22.978672 28575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:22.979009 28573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39e48ed968c749caac18330262f12ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e48ed968c749caac18330262f12ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38523 } } }
I20260929 18:46:22.979144 28573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:22.979060 28584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:22.982556 28584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:22.983263 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:22.977864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49675 (local address 127.0.0.1:38523)
0929 18:46:22.977990 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:22.977994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:22.978240 (+ 246us) server_negotiation.cc:407] Connection header received
0929 18:46:22.978342 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:22.978345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:22.978401 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:22.978477 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:22.979556 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:22.980322 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:22.981415 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:22.981645 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:22.981733 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:22.982501 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:22.982505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:22.982507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:22.982521 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:22.982551 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:22.982564 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:22.982621 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:22.983060 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 18:46:22.983135 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:46:22.988323 28584 catalog_manager.cc:1347] Generated new cluster ID: 4fc21f523fc34a119b5bdabe8589fe49
I20260929 18:46:22.988404 28584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:23.026106 28584 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:23.027100 28584 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:23.033228 28584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39e48ed968c749caac18330262f12ef5: Generated new TSK 0
I20260929 18:46:23.033901 28584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:23.044845 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 28499
2026-09-29T18:46:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:23 dist-test-slave-smpn krb5kdc[28598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:23 dist-test-slave-smpn krb5kdc[28598](info): set up 2 sockets
Sep 29 18:46:23 dist-test-slave-smpn krb5kdc[28598](info): commencing operation
krb5kdc: starting...
W20260929 18:46:23.991161 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:24 dist-test-slave-smpn krb5kdc[28598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:46:24.103225 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:24.212352 28611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:24.212626 28611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:24.212685 28611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:24.216741 28611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:24.216809 28611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:24.216826 28611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:24.216841 28611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:24.216868 28611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:24.222347 28611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:24.223487 28611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:24.224519 28611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:24.232766 28619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:24.232751 28616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:24.232756 28617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:24.232913 28611 server_base.cc:1034] running on GCE node
I20260929 18:46:24.233583 28611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:24.234727 28611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:24.235898 28611 hybrid_clock.cc:648] HybridClock initialized: now 1790707584235873 us; error 37 us; skew 500 ppm
Sep 29 18:46:24 dist-test-slave-smpn krb5kdc[28598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:24.238637 28611 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:46:24.239677 28611 webserver.cc:460] Webserver started at http://127.0.0.1:41721/ using document root <none> and password file <none>
I20260929 18:46:24.240239 28611 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:24.240311 28611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:24.240536 28611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:24.242669 28611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "709f1a3ca05147dea3158103c6c6a34c"
format_stamp: "Formatted at 2026-09-29 18:46:24 on dist-test-slave-smpn"
server_key: "d634524582249edcf1f47a4c0e5c7ef7"
server_key_iv: "6eb42611add549f0a14155d6f56f991d"
server_key_version: "encryptionkey@0"
I20260929 18:46:24.243273 28611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "709f1a3ca05147dea3158103c6c6a34c"
format_stamp: "Formatted at 2026-09-29 18:46:24 on dist-test-slave-smpn"
server_key: "d634524582249edcf1f47a4c0e5c7ef7"
server_key_iv: "6eb42611add549f0a14155d6f56f991d"
server_key_version: "encryptionkey@0"
I20260929 18:46:24.247234 28611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:46:24.249851 28626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:24.251230 28611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:24.251350 28611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "709f1a3ca05147dea3158103c6c6a34c"
format_stamp: "Formatted at 2026-09-29 18:46:24 on dist-test-slave-smpn"
server_key: "d634524582249edcf1f47a4c0e5c7ef7"
server_key_iv: "6eb42611add549f0a14155d6f56f991d"
server_key_version: "encryptionkey@0"
I20260929 18:46:24.251457 28611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:24.256006 28611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:24.258872 28611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:24.259037 28611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:24.265637 28611 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43005
I20260929 18:46:24.265645 28678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43005 every 8 connection(s)
I20260929 18:46:24.266566 28611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:24.269012 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 28611
I20260929 18:46:24.269146 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:24.269454 7797 external_mini_cluster.cc:1428] Setting key fc1e786fa80eb4f6dbde5066247654dd
I20260929 18:46:24.269531 28679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:24.274745 28682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:24.270788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42651 (local address 127.0.0.1:43005)
0929 18:46:24.271231 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:24.271248 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:24.271283 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:46:24.272166 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:24.272184 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:24.272490 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:24.272766 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:24.273578 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:24.273582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:24.273594 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:24.273603 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:24.273657 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:24.273677 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:24.273780 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:24.273881 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 18:46:24.274021 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":111,"threads_started":1}
I20260929 18:46:24.276346 28679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c: Bootstrap starting.
I20260929 18:46:24.277462 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:24.278990 28679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:24.279906 28679 log.cc:824] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:24.281281 28682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:24.279745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37793 (local address 127.0.0.1:43005)
0929 18:46:24.279875 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:24.279879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:24.280027 (+ 148us) server_negotiation.cc:407] Connection header received
0929 18:46:24.280137 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:24.280139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:24.280174 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:24.280261 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:24.280953 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:24.280957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:24.280959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:24.280971 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:24.280998 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:24.281010 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:24.281093 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:24.281139 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 18:46:24.281173 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:46:24.282100 28679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c: No bootstrap required, opened a new log
I20260929 18:46:24.285892 28679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709f1a3ca05147dea3158103c6c6a34c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43005 } }
I20260929 18:46:24.286114 28679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:24.286163 28679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709f1a3ca05147dea3158103c6c6a34c, State: Initialized, Role: FOLLOWER
I20260929 18:46:24.286547 28679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709f1a3ca05147dea3158103c6c6a34c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43005 } }
I20260929 18:46:24.286664 28679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:24.286710 28679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:24.286777 28679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:24.287744 28679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709f1a3ca05147dea3158103c6c6a34c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43005 } }
I20260929 18:46:24.288117 28679 leader_election.cc:302] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 709f1a3ca05147dea3158103c6c6a34c; no voters:
I20260929 18:46:24.288398 28679 leader_election.cc:288] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:24.288491 28689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:24.288691 28689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [term 1 LEADER]: Becoming Leader. State: Replica: 709f1a3ca05147dea3158103c6c6a34c, State: Running, Role: LEADER
I20260929 18:46:24.289042 28689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709f1a3ca05147dea3158103c6c6a34c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43005 } }
I20260929 18:46:24.289512 28679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:24.290521 28690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "709f1a3ca05147dea3158103c6c6a34c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709f1a3ca05147dea3158103c6c6a34c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43005 } } }
I20260929 18:46:24.290668 28690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:24.290951 28691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 709f1a3ca05147dea3158103c6c6a34c. Latest consensus state: current_term: 1 leader_uuid: "709f1a3ca05147dea3158103c6c6a34c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709f1a3ca05147dea3158103c6c6a34c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43005 } } }
I20260929 18:46:24.291034 28691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:24.291031 28698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:24.294142 28698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:24.300491 28698 catalog_manager.cc:1347] Generated new cluster ID: 9fff6beeb9ff4e0eb48d7fd3adaa80ea
I20260929 18:46:24.300554 28698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:24.312572 28698 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:24.312908 28698 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:24.322566 28698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 709f1a3ca05147dea3158103c6c6a34c: Generated new TSK 0
I20260929 18:46:24.323320 28698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:24.331645 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 28611
2026-09-29T18:46:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:24 dist-test-slave-smpn krb5kdc[28710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:24 dist-test-slave-smpn krb5kdc[28710](info): set up 2 sockets
Sep 29 18:46:24 dist-test-slave-smpn krb5kdc[28710](info): commencing operation
krb5kdc: starting...
W20260929 18:46:25.280238 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:25 dist-test-slave-smpn krb5kdc[28710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:46:25.397425 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:25.505277 28723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:25.505555 28723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:25.505595 28723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:25.509588 28723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:25.509651 28723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:25.509670 28723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:25.509684 28723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:25.509698 28723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:25.514956 28723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:25.515997 28723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:25.517004 28723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:25.525228 28728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:25.525245 28731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:25.525245 28729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:25.525297 28723 server_base.cc:1034] running on GCE node
I20260929 18:46:25.525980 28723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:25.527122 28723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:25.528285 28723 hybrid_clock.cc:648] HybridClock initialized: now 1790707585528268 us; error 29 us; skew 500 ppm
Sep 29 18:46:25 dist-test-slave-smpn krb5kdc[28710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:25.530866 28723 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:46:25.531791 28723 webserver.cc:460] Webserver started at http://127.0.0.1:37157/ using document root <none> and password file <none>
I20260929 18:46:25.532346 28723 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:25.532413 28723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:25.532618 28723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:25.534646 28723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "569e5248941a41a694fdf0473fd5f66c"
format_stamp: "Formatted at 2026-09-29 18:46:25 on dist-test-slave-smpn"
server_key: "e66ad3004a80711f922de5e161486da8"
server_key_iv: "6cbdb579d050ccac0cbe6f85d030e9ad"
server_key_version: "encryptionkey@0"
I20260929 18:46:25.535312 28723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "569e5248941a41a694fdf0473fd5f66c"
format_stamp: "Formatted at 2026-09-29 18:46:25 on dist-test-slave-smpn"
server_key: "e66ad3004a80711f922de5e161486da8"
server_key_iv: "6cbdb579d050ccac0cbe6f85d030e9ad"
server_key_version: "encryptionkey@0"
I20260929 18:46:25.539008 28723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 18:46:25.541450 28738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:25.542626 28723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:46:25.542737 28723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "569e5248941a41a694fdf0473fd5f66c"
format_stamp: "Formatted at 2026-09-29 18:46:25 on dist-test-slave-smpn"
server_key: "e66ad3004a80711f922de5e161486da8"
server_key_iv: "6cbdb579d050ccac0cbe6f85d030e9ad"
server_key_version: "encryptionkey@0"
I20260929 18:46:25.542832 28723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:25.568650 28723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:25.571802 28723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:25.571987 28723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:25.578711 28723 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34177
I20260929 18:46:25.578724 28790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34177 every 8 connection(s)
I20260929 18:46:25.579706 28723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:25.582468 28791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:25.583477 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 28723
I20260929 18:46:25.583596 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:25.583856 7797 external_mini_cluster.cc:1428] Setting key cc40f92a60aa5b35b807cfcb4b624782
I20260929 18:46:25.588660 28791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c: Bootstrap starting.
I20260929 18:46:25.590876 28791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:25.591614 28791 log.cc:824] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:46:25 dist-test-slave-smpn krb5kdc[28710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:46:25.593604 28791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c: No bootstrap required, opened a new log
I20260929 18:46:25.597270 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:25.585228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:49583 (local address 127.0.0.1:34177)
0929 18:46:25.585687 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:25.585704 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:25.585741 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:46:25.586414 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:25.586432 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:25.586796 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:25.587141 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:25.587923 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:25.588948 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:25.589774 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:25.590264 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:25.590416 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:25.592955 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:25.592960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:25.592973 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:25.592999 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:25.594490 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:25.595002 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:25.595006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:25.595010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:25.595154 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:25.595490 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:25.595492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:25.595493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:25.595845 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:25.595994 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:25.596313 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 18:46:25.596523 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":135,"threads_started":1}
I20260929 18:46:25.597455 28791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e5248941a41a694fdf0473fd5f66c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34177 } }
I20260929 18:46:25.597702 28791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:25.597759 28791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569e5248941a41a694fdf0473fd5f66c, State: Initialized, Role: FOLLOWER
I20260929 18:46:25.598156 28791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e5248941a41a694fdf0473fd5f66c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34177 } }
I20260929 18:46:25.598281 28791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:25.598342 28791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:25.598439 28791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:25.599373 28791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e5248941a41a694fdf0473fd5f66c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34177 } }
I20260929 18:46:25.599705 28791 leader_election.cc:302] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 569e5248941a41a694fdf0473fd5f66c; no voters:
I20260929 18:46:25.599992 28791 leader_election.cc:288] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:25.600052 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:25.600081 28796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:25.600301 28796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [term 1 LEADER]: Becoming Leader. State: Replica: 569e5248941a41a694fdf0473fd5f66c, State: Running, Role: LEADER
I20260929 18:46:25.600652 28796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e5248941a41a694fdf0473fd5f66c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34177 } }
I20260929 18:46:25.601234 28791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:25.601801 28796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 569e5248941a41a694fdf0473fd5f66c. Latest consensus state: current_term: 1 leader_uuid: "569e5248941a41a694fdf0473fd5f66c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e5248941a41a694fdf0473fd5f66c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34177 } } }
I20260929 18:46:25.601979 28796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:25.602186 28797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "569e5248941a41a694fdf0473fd5f66c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e5248941a41a694fdf0473fd5f66c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34177 } } }
I20260929 18:46:25.602293 28797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:25.602392 28807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:25.605463 28807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:25.611560 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:25.602032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:52249 (local address 127.0.0.1:34177)
0929 18:46:25.602139 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:25.602143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:25.603918 (+ 1775us) server_negotiation.cc:407] Connection header received
0929 18:46:25.604250 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:25.604253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:25.604297 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:25.604359 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:25.605070 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:25.605583 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:25.606650 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:25.606893 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:25.606974 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:25.608299 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:25.608303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:25.608306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:25.608330 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:25.609664 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:25.610198 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:25.610201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:25.610203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:25.610285 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:25.610643 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:25.610647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:25.610648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:25.610865 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:25.610998 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:25.611298 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 18:46:25.611402 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:46:25.611569 28807 catalog_manager.cc:1347] Generated new cluster ID: 9e11e45df60c48dbaa58040a37e18db7
I20260929 18:46:25.611646 28807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:25.634476 28807 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:25.635439 28807 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:25.645159 28807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 569e5248941a41a694fdf0473fd5f66c: Generated new TSK 0
I20260929 18:46:25.645854 28807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:25.654552 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 28723
W20260929 18:46:25.659086 28708 connection.cc:535] client connection to 127.0.0.1:34177 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34177): Connection reset by peer (error 104)
2026-09-29T18:46:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:25 dist-test-slave-smpn krb5kdc[28822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:25 dist-test-slave-smpn krb5kdc[28822](info): set up 2 sockets
Sep 29 18:46:25 dist-test-slave-smpn krb5kdc[28822](info): commencing operation
krb5kdc: starting...
W20260929 18:46:26.606997 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:26 dist-test-slave-smpn krb5kdc[28822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:46:26.717579 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:26.826608 28835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:26.826874 28835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:26.826915 28835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:26.830909 28835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:26.830971 28835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:26.830989 28835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:26.831004 28835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:26.831022 28835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:26.836439 28835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:26.837499 28835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:26.838533 28835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:26.846834 28841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:26.846820 28840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:26.846817 28835 server_base.cc:1034] running on GCE node
W20260929 18:46:26.846840 28843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:26.847513 28835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:26.848553 28835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:26.849707 28835 hybrid_clock.cc:648] HybridClock initialized: now 1790707586849692 us; error 27 us; skew 500 ppm
Sep 29 18:46:26 dist-test-slave-smpn krb5kdc[28822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:26.857623 28835 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:46:26.858512 28835 webserver.cc:460] Webserver started at http://127.0.0.1:46433/ using document root <none> and password file <none>
I20260929 18:46:26.859043 28835 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:26.859136 28835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:26.859314 28835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:26.861317 28835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "707c0e54a78141f69cfc0907706e1b10"
format_stamp: "Formatted at 2026-09-29 18:46:26 on dist-test-slave-smpn"
server_key: "f0e87e5e015f4e1785404dec77c0845d"
server_key_iv: "5b51e30d1674fd3c8eef99baab8a64f0"
server_key_version: "encryptionkey@0"
I20260929 18:46:26.861799 28835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "707c0e54a78141f69cfc0907706e1b10"
format_stamp: "Formatted at 2026-09-29 18:46:26 on dist-test-slave-smpn"
server_key: "f0e87e5e015f4e1785404dec77c0845d"
server_key_iv: "5b51e30d1674fd3c8eef99baab8a64f0"
server_key_version: "encryptionkey@0"
I20260929 18:46:26.865273 28835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:46:26.867530 28850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:26.868687 28835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:26.868790 28835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "707c0e54a78141f69cfc0907706e1b10"
format_stamp: "Formatted at 2026-09-29 18:46:26 on dist-test-slave-smpn"
server_key: "f0e87e5e015f4e1785404dec77c0845d"
server_key_iv: "5b51e30d1674fd3c8eef99baab8a64f0"
server_key_version: "encryptionkey@0"
I20260929 18:46:26.868899 28835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:26.896700 28835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:26.900033 28835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:26.900226 28835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:26.907583 28902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43193 every 8 connection(s)
I20260929 18:46:26.907549 28835 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43193
I20260929 18:46:26.908658 28835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:26.911495 28903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:26.913437 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 28835
I20260929 18:46:26.913573 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:26.913851 7797 external_mini_cluster.cc:1428] Setting key dac254742b75643daf6a67c65deaae77
I20260929 18:46:26.918069 28903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10: Bootstrap starting.
I20260929 18:46:26.920665 28903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:26.921456 28903 log.cc:824] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:26.922806 28906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:26.915254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:43669 (local address 127.0.0.1:43193)
0929 18:46:26.915660 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:26.915677 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:26.915714 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:46:26.916422 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:26.916445 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:26.916813 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:26.917104 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:26.917424 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:26.918407 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:26.919561 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:26.920045 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:26.920184 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:26.921107 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:26.921112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:26.921125 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:26.921521 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:26.921588 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:26.921619 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:26.921732 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:26.921943 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 18:46:26.922078 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":120,"threads_started":1}
I20260929 18:46:26.923384 28903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10: No bootstrap required, opened a new log
I20260929 18:46:26.925585 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:26.927425 28903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707c0e54a78141f69cfc0907706e1b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43193 } }
I20260929 18:46:26.927706 28903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:26.927799 28903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 707c0e54a78141f69cfc0907706e1b10, State: Initialized, Role: FOLLOWER
I20260929 18:46:26.928309 28903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707c0e54a78141f69cfc0907706e1b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43193 } }
I20260929 18:46:26.928481 28903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:26.928551 28903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:26.928653 28903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:26.929555 28903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707c0e54a78141f69cfc0907706e1b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43193 } }
I20260929 18:46:26.929883 28903 leader_election.cc:302] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 707c0e54a78141f69cfc0907706e1b10; no voters:
I20260929 18:46:26.930169 28903 leader_election.cc:288] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:26.930245 28913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:26.930462 28913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [term 1 LEADER]: Becoming Leader. State: Replica: 707c0e54a78141f69cfc0907706e1b10, State: Running, Role: LEADER
I20260929 18:46:26.930842 28913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707c0e54a78141f69cfc0907706e1b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43193 } }
I20260929 18:46:26.931595 28903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:26.932005 28906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:26.927868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:49623 (local address 127.0.0.1:43193)
0929 18:46:26.927981 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:26.927985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:26.928117 (+ 132us) server_negotiation.cc:407] Connection header received
0929 18:46:26.928218 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:26.928221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:26.928267 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:26.928360 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:26.928637 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:26.929186 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:26.930072 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:26.930434 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:26.930531 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:26.931533 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:26.931536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:26.931539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:26.931552 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:26.931580 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:26.931592 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:26.931676 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:26.931839 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 18:46:26.931878 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:46:26.932838 28914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "707c0e54a78141f69cfc0907706e1b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707c0e54a78141f69cfc0907706e1b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43193 } } }
I20260929 18:46:26.932953 28914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:26.933297 28915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 707c0e54a78141f69cfc0907706e1b10. Latest consensus state: current_term: 1 leader_uuid: "707c0e54a78141f69cfc0907706e1b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707c0e54a78141f69cfc0907706e1b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43193 } } }
I20260929 18:46:26.933920 28915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:26.933960 28923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:26.936964 28923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:26.942019 28923 catalog_manager.cc:1347] Generated new cluster ID: b80d9a7590734da69e7d2d7796fd59b0
I20260929 18:46:26.942093 28923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:26.964274 28923 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:26.965250 28923 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:26.980134 28923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 707c0e54a78141f69cfc0907706e1b10: Generated new TSK 0
I20260929 18:46:26.980821 28923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:26.990270 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 28835
W20260929 18:46:26.994913 28820 connection.cc:535] client connection to 127.0.0.1:43193 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43193): Connection reset by peer (error 104)
2026-09-29T18:46:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:27 dist-test-slave-smpn krb5kdc[28934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:27 dist-test-slave-smpn krb5kdc[28934](info): set up 2 sockets
Sep 29 18:46:27 dist-test-slave-smpn krb5kdc[28934](info): commencing operation
krb5kdc: starting...
W20260929 18:46:27.942977 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:28 dist-test-slave-smpn krb5kdc[28934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:46:28.055361 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:28.161700 28947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:28.161965 28947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:28.162020 28947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:28.165931 28947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:28.165992 28947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:28.166018 28947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:28.166038 28947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:28.166059 28947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:28.171155 28947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:28.172120 28947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:28.173092 28947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:46:28.181499 28947 server_base.cc:1034] running on GCE node
W20260929 18:46:28.181456 28952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:28.181452 28953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:28.181452 28955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:28.182305 28947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:28.183461 28947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:28.184620 28947 hybrid_clock.cc:648] HybridClock initialized: now 1790707588184598 us; error 36 us; skew 500 ppm
Sep 29 18:46:28 dist-test-slave-smpn krb5kdc[28934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:28.187269 28947 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:46:28.188180 28947 webserver.cc:460] Webserver started at http://127.0.0.1:38071/ using document root <none> and password file <none>
I20260929 18:46:28.188728 28947 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:28.188794 28947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:28.189014 28947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:28.191130 28947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "33fddb7157204bd3aad33fcb5b2582ea"
format_stamp: "Formatted at 2026-09-29 18:46:28 on dist-test-slave-smpn"
server_key: "428d2aa8e34006a691e9529471aecf36"
server_key_iv: "8bc8cf94729a45aa8d797be3a86e50b5"
server_key_version: "encryptionkey@0"
I20260929 18:46:28.191685 28947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "33fddb7157204bd3aad33fcb5b2582ea"
format_stamp: "Formatted at 2026-09-29 18:46:28 on dist-test-slave-smpn"
server_key: "428d2aa8e34006a691e9529471aecf36"
server_key_iv: "8bc8cf94729a45aa8d797be3a86e50b5"
server_key_version: "encryptionkey@0"
I20260929 18:46:28.195327 28947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:46:28.197649 28962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:28.198797 28947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:46:28.198913 28947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "33fddb7157204bd3aad33fcb5b2582ea"
format_stamp: "Formatted at 2026-09-29 18:46:28 on dist-test-slave-smpn"
server_key: "428d2aa8e34006a691e9529471aecf36"
server_key_iv: "8bc8cf94729a45aa8d797be3a86e50b5"
server_key_version: "encryptionkey@0"
I20260929 18:46:28.199009 28947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:28.203529 28947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:28.206207 28947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:28.206375 28947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:28.213227 28947 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40169
I20260929 18:46:28.213256 29014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40169 every 8 connection(s)
I20260929 18:46:28.214118 28947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:28.216773 29015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:28.220293 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 28947
I20260929 18:46:28.220430 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:28.220674 7797 external_mini_cluster.cc:1428] Setting key 68a70082c96a2c8cbbc378be5b84e51c
I20260929 18:46:28.222468 29015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea: Bootstrap starting.
I20260929 18:46:28.224694 29015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:28.225466 29015 log.cc:824] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:28.225780 29018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:28.222002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:50849 (local address 127.0.0.1:40169)
0929 18:46:28.222437 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:28.222452 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:28.222486 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:46:28.223151 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:28.223170 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:28.223461 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:28.223769 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:28.224509 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:28.224514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:28.224525 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:28.224535 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:28.224587 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:28.224606 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:28.224697 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:28.224816 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 18:46:28.224983 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":129,"threads_started":1}
I20260929 18:46:28.228142 29015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea: No bootstrap required, opened a new log
I20260929 18:46:28.228327 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:28.232126 29018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:28.230501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:38157 (local address 127.0.0.1:40169)
0929 18:46:28.230619 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:28.230623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:28.230830 (+ 207us) server_negotiation.cc:407] Connection header received
0929 18:46:28.230935 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:28.230938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:28.230972 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:28.231035 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:28.231759 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:28.231763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:28.231766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:28.231777 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:28.231803 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:28.231822 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:28.231893 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:28.231968 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 18:46:28.232008 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:46:28.232486 29015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fddb7157204bd3aad33fcb5b2582ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } }
I20260929 18:46:28.232728 29015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:28.232774 29015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33fddb7157204bd3aad33fcb5b2582ea, State: Initialized, Role: FOLLOWER
I20260929 18:46:28.233215 29015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fddb7157204bd3aad33fcb5b2582ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } }
I20260929 18:46:28.233374 29015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:28.233438 29015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:28.233532 29015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:28.234417 29015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fddb7157204bd3aad33fcb5b2582ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } }
I20260929 18:46:28.234745 29015 leader_election.cc:302] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33fddb7157204bd3aad33fcb5b2582ea; no voters:
I20260929 18:46:28.235020 29015 leader_election.cc:288] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:28.235154 29025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:28.235349 29025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [term 1 LEADER]: Becoming Leader. State: Replica: 33fddb7157204bd3aad33fcb5b2582ea, State: Running, Role: LEADER
I20260929 18:46:28.235718 29025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fddb7157204bd3aad33fcb5b2582ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } }
I20260929 18:46:28.236321 29015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:28.237346 29026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33fddb7157204bd3aad33fcb5b2582ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fddb7157204bd3aad33fcb5b2582ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } } }
I20260929 18:46:28.237452 29026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:28.237716 29027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33fddb7157204bd3aad33fcb5b2582ea. Latest consensus state: current_term: 1 leader_uuid: "33fddb7157204bd3aad33fcb5b2582ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33fddb7157204bd3aad33fcb5b2582ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } } }
I20260929 18:46:28.237797 29027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:28.237829 29034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:28.240998 29034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:28.246627 29034 catalog_manager.cc:1347] Generated new cluster ID: d75aa730aaa4433b9a77444da8ae7d44
I20260929 18:46:28.246690 29034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:28.255990 29034 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:28.256273 29034 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:28.263602 29034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33fddb7157204bd3aad33fcb5b2582ea: Generated new TSK 0
I20260929 18:46:28.264389 29034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:28.275803 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 28947
2026-09-29T18:46:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:28 dist-test-slave-smpn krb5kdc[29046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:28 dist-test-slave-smpn krb5kdc[29046](info): set up 2 sockets
Sep 29 18:46:28 dist-test-slave-smpn krb5kdc[29046](info): commencing operation
krb5kdc: starting...
W20260929 18:46:29.226979 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:29 dist-test-slave-smpn krb5kdc[29046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:46:29.338078 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43007
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:29.444262 29059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:29.444547 29059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:29.444613 29059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:29.448681 29059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:29.448748 29059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:29.448779 29059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:29.448800 29059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:29.448820 29059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:29.454138 29059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43007
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:29.455248 29059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:29.456252 29059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:29.464280 29065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:29.464273 29067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:29.464448 29059 server_base.cc:1034] running on GCE node
W20260929 18:46:29.464342 29064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:29.465055 29059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:29.466089 29059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:29.467239 29059 hybrid_clock.cc:648] HybridClock initialized: now 1790707589467229 us; error 40 us; skew 500 ppm
Sep 29 18:46:29 dist-test-slave-smpn krb5kdc[29046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:29.469842 29059 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:46:29.470696 29059 webserver.cc:460] Webserver started at http://127.0.0.1:40029/ using document root <none> and password file <none>
I20260929 18:46:29.471280 29059 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:29.471354 29059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:29.471556 29059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:29.473590 29059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "d83582a861a34bb18b7fa13639576e25"
format_stamp: "Formatted at 2026-09-29 18:46:29 on dist-test-slave-smpn"
server_key: "49634129f5df911481417dc65be46092"
server_key_iv: "c54e0b7e5f8263316b0660b88858599d"
server_key_version: "encryptionkey@0"
I20260929 18:46:29.474123 29059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "d83582a861a34bb18b7fa13639576e25"
format_stamp: "Formatted at 2026-09-29 18:46:29 on dist-test-slave-smpn"
server_key: "49634129f5df911481417dc65be46092"
server_key_iv: "c54e0b7e5f8263316b0660b88858599d"
server_key_version: "encryptionkey@0"
I20260929 18:46:29.477656 29059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 18:46:29.479947 29074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:29.481086 29059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:29.481196 29059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "d83582a861a34bb18b7fa13639576e25"
format_stamp: "Formatted at 2026-09-29 18:46:29 on dist-test-slave-smpn"
server_key: "49634129f5df911481417dc65be46092"
server_key_iv: "c54e0b7e5f8263316b0660b88858599d"
server_key_version: "encryptionkey@0"
I20260929 18:46:29.481290 29059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:29.495566 29059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:29.498293 29059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:29.498468 29059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:29.505013 29059 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43007
I20260929 18:46:29.505019 29126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43007 every 8 connection(s)
I20260929 18:46:29.506172 29059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:29.509107 29127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:29.513187 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 29059
I20260929 18:46:29.513327 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:29.513569 7797 external_mini_cluster.cc:1428] Setting key 63496b03dff5bb3eab6b57ec71ce4ab8
I20260929 18:46:29.514731 29127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25: Bootstrap starting.
I20260929 18:46:29.516978 29127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:29.517769 29127 log.cc:824] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:29.519847 29127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25: No bootstrap required, opened a new log
Sep 29 18:46:29 dist-test-slave-smpn krb5kdc[29046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:46:29.524068 29127 raft_consensus.cc:348] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83582a861a34bb18b7fa13639576e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43007 } }
I20260929 18:46:29.524353 29127 raft_consensus.cc:374] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:29.524408 29127 raft_consensus.cc:729] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d83582a861a34bb18b7fa13639576e25, State: Initialized, Role: FOLLOWER
I20260929 18:46:29.524879 29127 consensus_queue.cc:260] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83582a861a34bb18b7fa13639576e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43007 } }
I20260929 18:46:29.525038 29127 raft_consensus.cc:388] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:29.525103 29127 raft_consensus.cc:482] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:29.525202 29127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:29.526005 29127 raft_consensus.cc:504] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83582a861a34bb18b7fa13639576e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43007 } }
I20260929 18:46:29.526317 29127 leader_election.cc:302] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d83582a861a34bb18b7fa13639576e25; no voters:
I20260929 18:46:29.526636 29127 leader_election.cc:288] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:29.526690 29132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:29.526899 29132 raft_consensus.cc:686] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [term 1 LEADER]: Becoming Leader. State: Replica: d83582a861a34bb18b7fa13639576e25, State: Running, Role: LEADER
I20260929 18:46:29.527251 29132 consensus_queue.cc:237] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83582a861a34bb18b7fa13639576e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43007 } }
I20260929 18:46:29.527865 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:29.514912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:47603 (local address 127.0.0.1:43007)
0929 18:46:29.515316 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:29.515333 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:29.515368 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:46:29.515991 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:29.516007 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:29.516343 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:29.516613 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:29.517452 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:29.518616 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:29.519739 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:29.520229 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:29.520358 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:29.523026 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:29.523032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:29.523049 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:29.523104 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:29.524861 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:29.525375 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:29.525382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:29.525388 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:29.525496 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:29.525803 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:29.525806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:29.525809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:29.526234 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:29.526436 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:29.526749 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 18:46:29.526978 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":131,"threads_started":1}
I20260929 18:46:29.527925 29127 sys_catalog.cc:564] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:29.528869 29133 sys_catalog.cc:455] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d83582a861a34bb18b7fa13639576e25. Latest consensus state: current_term: 1 leader_uuid: "d83582a861a34bb18b7fa13639576e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83582a861a34bb18b7fa13639576e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43007 } } }
I20260929 18:46:29.528987 29133 sys_catalog.cc:458] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:29.529188 29133 sys_catalog.cc:455] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d83582a861a34bb18b7fa13639576e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83582a861a34bb18b7fa13639576e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43007 } } }
I20260929 18:46:29.529259 29133 sys_catalog.cc:458] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:29.529636 29141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:29.531036 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:29.532779 29141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:29.538373 29141 catalog_manager.cc:1347] Generated new cluster ID: 691668f837a74c8486e163735c72d6b4
I20260929 18:46:29.538434 29141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:29.542930 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:29.535060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:48501 (local address 127.0.0.1:43007)
0929 18:46:29.535183 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:29.535187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:29.535550 (+ 363us) server_negotiation.cc:407] Connection header received
0929 18:46:29.535657 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:29.535660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:29.535720 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:29.535801 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:29.536526 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:29.537362 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:29.538440 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:29.538702 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:29.538794 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:29.540085 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:29.540089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:29.540091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:29.540112 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:29.541377 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:29.541803 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:29.541806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:29.541808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:29.541878 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:29.542170 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:29.542173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:29.542175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:29.542341 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:29.542472 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:29.542708 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 18:46:29.542790 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:46:29.551416 29141 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:29.552294 29141 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:29.563812 29141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d83582a861a34bb18b7fa13639576e25: Generated new TSK 0
I20260929 18:46:29.564527 29141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:29.568281 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 29059
W20260929 18:46:29.573063 29044 connection.cc:535] client connection to 127.0.0.1:43007 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43007): Connection reset by peer (error 104)
2026-09-29T18:46:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:29 dist-test-slave-smpn krb5kdc[29158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:29 dist-test-slave-smpn krb5kdc[29158](info): set up 2 sockets
Sep 29 18:46:29 dist-test-slave-smpn krb5kdc[29158](info): commencing operation
krb5kdc: starting...
W20260929 18:46:30.514986 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:30 dist-test-slave-smpn krb5kdc[29158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:46:30.628211 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32883
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:30.736186 29171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:30.736449 29171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:30.736491 29171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:30.740423 29171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:30.740478 29171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:30.740496 29171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:30.740511 29171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:30.740525 29171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:30.745777 29171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32883
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:30.746821 29171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:30.747830 29171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:30.755815 29176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:30.755898 29171 server_base.cc:1034] running on GCE node
W20260929 18:46:30.755823 29179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:30.755815 29177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:30.756580 29171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:30.757594 29171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:30.758744 29171 hybrid_clock.cc:648] HybridClock initialized: now 1790707590758728 us; error 30 us; skew 500 ppm
Sep 29 18:46:30 dist-test-slave-smpn krb5kdc[29158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:30.761338 29171 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:46:30.762158 29171 webserver.cc:460] Webserver started at http://127.0.0.1:38175/ using document root <none> and password file <none>
I20260929 18:46:30.762656 29171 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:30.762715 29171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:30.762943 29171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:30.764930 29171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "21c5063947b9439cafeb3c2007855b10"
format_stamp: "Formatted at 2026-09-29 18:46:30 on dist-test-slave-smpn"
server_key: "8d27fe56752a6b07bd820aa5657da428"
server_key_iv: "4f5fed44a9daeeb2aaf902d8b445a0b4"
server_key_version: "encryptionkey@0"
I20260929 18:46:30.765447 29171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "21c5063947b9439cafeb3c2007855b10"
format_stamp: "Formatted at 2026-09-29 18:46:30 on dist-test-slave-smpn"
server_key: "8d27fe56752a6b07bd820aa5657da428"
server_key_iv: "4f5fed44a9daeeb2aaf902d8b445a0b4"
server_key_version: "encryptionkey@0"
I20260929 18:46:30.769055 29171 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:46:30.771298 29186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:30.772439 29171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:30.772540 29171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "21c5063947b9439cafeb3c2007855b10"
format_stamp: "Formatted at 2026-09-29 18:46:30 on dist-test-slave-smpn"
server_key: "8d27fe56752a6b07bd820aa5657da428"
server_key_iv: "4f5fed44a9daeeb2aaf902d8b445a0b4"
server_key_version: "encryptionkey@0"
I20260929 18:46:30.772634 29171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:30.787982 29171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:30.790606 29171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:30.790782 29171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:30.797603 29171 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32883
I20260929 18:46:30.797616 29238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32883 every 8 connection(s)
I20260929 18:46:30.798554 29171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:30.801287 29239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:30.803602 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 29171
I20260929 18:46:30.803717 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:30.803958 7797 external_mini_cluster.cc:1428] Setting key a70dd47c5f00412d97a8208f4f578e02
I20260929 18:46:30.807437 29239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10: Bootstrap starting.
I20260929 18:46:30.809984 29239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:30.810853 29239 log.cc:824] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:30.812906 29239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10: No bootstrap required, opened a new log
I20260929 18:46:30.813508 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:30.805297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:48645 (local address 127.0.0.1:32883)
0929 18:46:30.805691 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:30.805709 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:30.805750 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:46:30.806541 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:30.806564 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:30.806988 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:30.807374 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:30.807778 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:30.809230 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:30.810083 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:30.810770 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:30.810924 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:30.811764 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:30.811768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:30.811781 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:30.812202 (+ 421us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:30.812255 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:30.812279 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:30.812406 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:30.812590 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 18:46:30.812725 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":93,"threads_started":1}
I20260929 18:46:30.816435 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:30.816864 29239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c5063947b9439cafeb3c2007855b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32883 } }
I20260929 18:46:30.817083 29239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:30.817137 29239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c5063947b9439cafeb3c2007855b10, State: Initialized, Role: FOLLOWER
I20260929 18:46:30.817530 29239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c5063947b9439cafeb3c2007855b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32883 } }
I20260929 18:46:30.817675 29239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:30.817739 29239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:30.817837 29239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:30.818843 29239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c5063947b9439cafeb3c2007855b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32883 } }
I20260929 18:46:30.819240 29239 leader_election.cc:302] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21c5063947b9439cafeb3c2007855b10; no voters:
I20260929 18:46:30.819567 29239 leader_election.cc:288] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:30.819676 29249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:30.819880 29249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [term 1 LEADER]: Becoming Leader. State: Replica: 21c5063947b9439cafeb3c2007855b10, State: Running, Role: LEADER
I20260929 18:46:30.820227 29249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c5063947b9439cafeb3c2007855b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32883 } }
I20260929 18:46:30.820868 29239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:30.821539 29249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21c5063947b9439cafeb3c2007855b10. Latest consensus state: current_term: 1 leader_uuid: "21c5063947b9439cafeb3c2007855b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c5063947b9439cafeb3c2007855b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32883 } } }
I20260929 18:46:30.821662 29249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:30.822297 29254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:30.824049 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:30.818370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:36091 (local address 127.0.0.1:32883)
0929 18:46:30.818482 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:30.818487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:30.818626 (+ 139us) server_negotiation.cc:407] Connection header received
0929 18:46:30.818711 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:30.818714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:30.818762 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:30.818851 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:30.819100 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:30.819861 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:30.821185 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:30.821537 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:30.821634 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:30.822910 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:30.822914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:30.822917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:30.822929 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:30.822956 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:30.822970 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:30.823041 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:30.823887 (+ 846us) server_negotiation.cc:299] Negotiation successful
0929 18:46:30.823923 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:46:30.825367 29250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21c5063947b9439cafeb3c2007855b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c5063947b9439cafeb3c2007855b10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32883 } } }
I20260929 18:46:30.825470 29250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:30.826530 29254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:30.832110 29254 catalog_manager.cc:1347] Generated new cluster ID: 646985f0d5d145c39da076fde8186c7a
I20260929 18:46:30.832165 29254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:30.858389 29254 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:30.859690 29254 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:30.865422 29254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21c5063947b9439cafeb3c2007855b10: Generated new TSK 0
I20260929 18:46:30.866254 29254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:30.870554 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 29171
W20260929 18:46:30.875255 29156 connection.cc:535] client connection to 127.0.0.1:32883 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32883): Connection reset by peer (error 104)
2026-09-29T18:46:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:30 dist-test-slave-smpn krb5kdc[29270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:30 dist-test-slave-smpn krb5kdc[29270](info): set up 2 sockets
Sep 29 18:46:30 dist-test-slave-smpn krb5kdc[29270](info): commencing operation
krb5kdc: starting...
W20260929 18:46:31.822992 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:31 dist-test-slave-smpn krb5kdc[29270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:46:31.935595 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37491
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:32.042852 29283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:32.043130 29283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:32.043174 29283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:32.047107 29283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:32.047163 29283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:32.047181 29283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:32.047196 29283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:32.047209 29283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:32.052862 29283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37491
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:32.053967 29283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:32.055011 29283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:32.063660 29291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:32.063620 29288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:32.063620 29289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:32.063728 29283 server_base.cc:1034] running on GCE node
I20260929 18:46:32.064415 29283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:32.065598 29283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:32.066751 29283 hybrid_clock.cc:648] HybridClock initialized: now 1790707592066725 us; error 42 us; skew 500 ppm
Sep 29 18:46:32 dist-test-slave-smpn krb5kdc[29270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:32.076519 29283 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:46:32.077577 29283 webserver.cc:460] Webserver started at http://127.0.0.1:44671/ using document root <none> and password file <none>
I20260929 18:46:32.078173 29283 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:32.078249 29283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:32.078487 29283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:32.080780 29283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "00d0aea5d8074f718bc6667bdacbacb1"
format_stamp: "Formatted at 2026-09-29 18:46:32 on dist-test-slave-smpn"
server_key: "d9fe6cced7a7c3fe19aca402698178fb"
server_key_iv: "f24952cfe6b986e20ab97702ddddfa96"
server_key_version: "encryptionkey@0"
I20260929 18:46:32.081384 29283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "00d0aea5d8074f718bc6667bdacbacb1"
format_stamp: "Formatted at 2026-09-29 18:46:32 on dist-test-slave-smpn"
server_key: "d9fe6cced7a7c3fe19aca402698178fb"
server_key_iv: "f24952cfe6b986e20ab97702ddddfa96"
server_key_version: "encryptionkey@0"
I20260929 18:46:32.085191 29283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:46:32.087603 29298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:32.088847 29283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:32.088968 29283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "00d0aea5d8074f718bc6667bdacbacb1"
format_stamp: "Formatted at 2026-09-29 18:46:32 on dist-test-slave-smpn"
server_key: "d9fe6cced7a7c3fe19aca402698178fb"
server_key_iv: "f24952cfe6b986e20ab97702ddddfa96"
server_key_version: "encryptionkey@0"
I20260929 18:46:32.089066 29283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:32.093565 29283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:32.096369 29283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:32.096524 29283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:32.103341 29283 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37491
I20260929 18:46:32.103398 29350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37491 every 8 connection(s)
I20260929 18:46:32.104228 29283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:32.106958 29351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:32.111403 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 29283
I20260929 18:46:32.111521 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:32.111775 7797 external_mini_cluster.cc:1428] Setting key f3d446e4fd8de9d433868e2843ab52d1
I20260929 18:46:32.113116 29351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1: Bootstrap starting.
I20260929 18:46:32.115516 29351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:32.116423 29351 log.cc:824] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:32.116992 29354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:32.113217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:56283 (local address 127.0.0.1:37491)
0929 18:46:32.113646 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:32.113660 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:32.113697 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:46:32.114351 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:32.114368 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:32.114661 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:32.114951 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:32.115770 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:32.115775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:32.115787 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:32.115798 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:32.115852 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:32.115872 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:32.115958 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:32.116068 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 18:46:32.116223 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
I20260929 18:46:32.118304 29351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1: No bootstrap required, opened a new log
I20260929 18:46:32.119678 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:32.122045 29351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0aea5d8074f718bc6667bdacbacb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37491 } }
I20260929 18:46:32.122292 29351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:32.122351 29351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d0aea5d8074f718bc6667bdacbacb1, State: Initialized, Role: FOLLOWER
I20260929 18:46:32.122812 29351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0aea5d8074f718bc6667bdacbacb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37491 } }
I20260929 18:46:32.122968 29351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:32.123042 29351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:32.123201 29351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:32.123297 29354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:32.121577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:43785 (local address 127.0.0.1:37491)
0929 18:46:32.121698 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:32.121702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:32.121900 (+ 198us) server_negotiation.cc:407] Connection header received
0929 18:46:32.122004 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:32.122007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:32.122038 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:32.122099 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:32.122780 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:32.122783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:32.122785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:32.122795 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:32.122820 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:32.122830 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:32.122895 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:32.123028 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 18:46:32.123139 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:46:32.124261 29351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0aea5d8074f718bc6667bdacbacb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37491 } }
I20260929 18:46:32.124590 29351 leader_election.cc:302] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00d0aea5d8074f718bc6667bdacbacb1; no voters:
I20260929 18:46:32.124886 29351 leader_election.cc:288] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:32.124970 29361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:32.125170 29361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [term 1 LEADER]: Becoming Leader. State: Replica: 00d0aea5d8074f718bc6667bdacbacb1, State: Running, Role: LEADER
I20260929 18:46:32.125494 29361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0aea5d8074f718bc6667bdacbacb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37491 } }
I20260929 18:46:32.126081 29351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:32.127025 29362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00d0aea5d8074f718bc6667bdacbacb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0aea5d8074f718bc6667bdacbacb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37491 } } }
I20260929 18:46:32.127295 29362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:32.127671 29370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:32.127660 29363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00d0aea5d8074f718bc6667bdacbacb1. Latest consensus state: current_term: 1 leader_uuid: "00d0aea5d8074f718bc6667bdacbacb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d0aea5d8074f718bc6667bdacbacb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37491 } } }
I20260929 18:46:32.127821 29363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:32.130607 29370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:32.135759 29370 catalog_manager.cc:1347] Generated new cluster ID: 592fc1fdbe814961883d107397399fc2
I20260929 18:46:32.135829 29370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:32.153049 29370 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:32.153369 29370 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:32.157068 29370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00d0aea5d8074f718bc6667bdacbacb1: Generated new TSK 0
I20260929 18:46:32.157859 29370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:32.169615 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 29283
2026-09-29T18:46:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:32 dist-test-slave-smpn krb5kdc[29382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:32 dist-test-slave-smpn krb5kdc[29382](info): set up 2 sockets
Sep 29 18:46:32 dist-test-slave-smpn krb5kdc[29382](info): commencing operation
krb5kdc: starting...
W20260929 18:46:33.123042 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:33 dist-test-slave-smpn krb5kdc[29382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:46:33.235387 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:32981
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:45261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:32981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:33.342514 29395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:33.342813 29395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:33.342875 29395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:33.346933 29395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:33.347018 29395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:33.347044 29395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:33.347084 29395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:33.347106 29395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:33.352687 29395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:45261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:32981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:33.353785 29395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:33.354837 29395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:33.363297 29400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:33.363377 29395 server_base.cc:1034] running on GCE node
W20260929 18:46:33.363337 29401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:33.363322 29403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:33.364212 29395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:33.365346 29395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:33.366499 29395 hybrid_clock.cc:648] HybridClock initialized: now 1790707593366482 us; error 31 us; skew 500 ppm
Sep 29 18:46:33 dist-test-slave-smpn krb5kdc[29382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707593, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:33.369189 29395 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:46:33.370096 29395 webserver.cc:460] Webserver started at http://127.7.157.126:39953/ using document root <none> and password file <none>
I20260929 18:46:33.370621 29395 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:33.370678 29395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:33.370849 29395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:33.372915 29395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "5a94e71e49c74fc5981c6ed3e1b1332d"
format_stamp: "Formatted at 2026-09-29 18:46:33 on dist-test-slave-smpn"
server_key: "045d8606360fdbd230ddcc869fd932a2"
server_key_iv: "a23f007dae51fe2efae7fc152296bcc8"
server_key_version: "encryptionkey@0"
I20260929 18:46:33.373472 29395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "5a94e71e49c74fc5981c6ed3e1b1332d"
format_stamp: "Formatted at 2026-09-29 18:46:33 on dist-test-slave-smpn"
server_key: "045d8606360fdbd230ddcc869fd932a2"
server_key_iv: "a23f007dae51fe2efae7fc152296bcc8"
server_key_version: "encryptionkey@0"
I20260929 18:46:33.377053 29395 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:46:33.379364 29410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:33.380628 29395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:46:33.380734 29395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "5a94e71e49c74fc5981c6ed3e1b1332d"
format_stamp: "Formatted at 2026-09-29 18:46:33 on dist-test-slave-smpn"
server_key: "045d8606360fdbd230ddcc869fd932a2"
server_key_iv: "a23f007dae51fe2efae7fc152296bcc8"
server_key_version: "encryptionkey@0"
I20260929 18:46:33.380838 29395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:33.407007 29395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:33.410239 29395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:33.410420 29395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:33.417465 29395 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:32981
I20260929 18:46:33.417485 29462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:32981 every 8 connection(s)
I20260929 18:46:33.418467 29395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:33.420904 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 29395
I20260929 18:46:33.421036 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:33.421335 7797 external_mini_cluster.cc:1428] Setting key 2e77ac2c1c25f1f81af7e6acb5f31888
I20260929 18:46:33.421514 29463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:33.428241 29463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d: Bootstrap starting.
Sep 29 18:46:33 dist-test-slave-smpn krb5kdc[29382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:33.430953 29463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:33.431679 29463 log.cc:824] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:33.433457 29463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d: No bootstrap required, opened a new log
I20260929 18:46:33.435787 29466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:33.422904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58743 (local address 127.7.157.126:32981)
0929 18:46:33.423361 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:33.423376 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:33.423410 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:46:33.424335 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:33.424354 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:33.424705 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:33.425014 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:33.425796 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:33.427244 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:33.428071 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:33.428575 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:33.428717 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:33.431390 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:33.431395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:33.431408 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:33.431434 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:33.432969 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:33.433472 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:33.433476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:33.433481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:33.433617 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:33.433920 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:33.433924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:33.433925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:33.434317 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:33.434516 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:33.434824 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 18:46:33.435028 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":167,"threads_started":1}
I20260929 18:46:33.437081 29463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a94e71e49c74fc5981c6ed3e1b1332d" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32981 } }
I20260929 18:46:33.437359 29463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:33.437414 29463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a94e71e49c74fc5981c6ed3e1b1332d, State: Initialized, Role: FOLLOWER
I20260929 18:46:33.437825 29463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a94e71e49c74fc5981c6ed3e1b1332d" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32981 } }
I20260929 18:46:33.437973 29463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:33.438042 29463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:33.438140 29463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:33.438689 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:33.439175 29463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a94e71e49c74fc5981c6ed3e1b1332d" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32981 } }
I20260929 18:46:33.439497 29463 leader_election.cc:302] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a94e71e49c74fc5981c6ed3e1b1332d; no voters:
I20260929 18:46:33.439771 29463 leader_election.cc:288] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:33.439865 29468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:33.440078 29468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [term 1 LEADER]: Becoming Leader. State: Replica: 5a94e71e49c74fc5981c6ed3e1b1332d, State: Running, Role: LEADER
I20260929 18:46:33.440430 29468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a94e71e49c74fc5981c6ed3e1b1332d" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32981 } }
I20260929 18:46:33.441077 29463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:33.441974 29473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a94e71e49c74fc5981c6ed3e1b1332d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a94e71e49c74fc5981c6ed3e1b1332d" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32981 } } }
I20260929 18:46:33.442152 29473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:33.442407 29473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a94e71e49c74fc5981c6ed3e1b1332d. Latest consensus state: current_term: 1 leader_uuid: "5a94e71e49c74fc5981c6ed3e1b1332d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a94e71e49c74fc5981c6ed3e1b1332d" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 32981 } } }
I20260929 18:46:33.442549 29473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:33.443351 29482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:33.446269 29482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:33.449119 29466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:33.440878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51015 (local address 127.7.157.126:32981)
0929 18:46:33.440996 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:33.441000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:33.441012 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:46:33.441044 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:33.441047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:33.441083 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:33.441145 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:33.441859 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:33.442611 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:33.443729 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:33.443960 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:33.444045 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:33.445545 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:33.445549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:33.445552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:33.445586 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:33.447247 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:33.447782 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:33.447785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:33.447787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:33.447876 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:33.448250 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:33.448253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:33.448255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:33.448454 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:33.448544 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:33.448886 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 18:46:33.448976 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:46:33.452086 29482 catalog_manager.cc:1347] Generated new cluster ID: 066b5ceb5fac49cfa0f7c747dd411c28
I20260929 18:46:33.452149 29482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:33.465626 29482 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:33.466902 29482 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:33.480959 29482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a94e71e49c74fc5981c6ed3e1b1332d: Generated new TSK 0
I20260929 18:46:33.481721 29482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:33.492267 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 29395
W20260929 18:46:33.496845 29380 connection.cc:535] client connection to 127.7.157.126:32981 recv error: Network error: failed to read from TLS socket (remote: 127.7.157.126:32981): Connection reset by peer (error 104)
2026-09-29T18:46:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:33 dist-test-slave-smpn krb5kdc[29494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:33 dist-test-slave-smpn krb5kdc[29494](info): set up 2 sockets
Sep 29 18:46:33 dist-test-slave-smpn krb5kdc[29494](info): commencing operation
krb5kdc: starting...
W20260929 18:46:34.438942 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:34 dist-test-slave-smpn krb5kdc[29494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:46:34.550402 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:39935
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:34347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:39935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:34.657814 29507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:34.658113 29507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:34.658171 29507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:34.662252 29507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:34.662333 29507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:34.662360 29507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:34.662380 29507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:34.662400 29507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:34.667789 29507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:34347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:39935
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:39935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:34.668900 29507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:34.669957 29507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:34.678428 29513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:34.678404 29512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:34.678460 29507 server_base.cc:1034] running on GCE node
W20260929 18:46:34.678408 29515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:34.679234 29507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:34.680371 29507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:34.681525 29507 hybrid_clock.cc:648] HybridClock initialized: now 1790707594681510 us; error 29 us; skew 500 ppm
Sep 29 18:46:34 dist-test-slave-smpn krb5kdc[29494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707594, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:34.684188 29507 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:46:34.685103 29507 webserver.cc:460] Webserver started at http://127.7.157.126:44927/ using document root <none> and password file <none>
I20260929 18:46:34.685647 29507 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:34.685710 29507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:34.685914 29507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:34.688028 29507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "2eb6a1eabc8449949778a487700be416"
format_stamp: "Formatted at 2026-09-29 18:46:34 on dist-test-slave-smpn"
server_key: "873bd3cc52dbe1f54337cb861979c935"
server_key_iv: "ced6f4fbfb3473f139afc526c9626baf"
server_key_version: "encryptionkey@0"
I20260929 18:46:34.688575 29507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "2eb6a1eabc8449949778a487700be416"
format_stamp: "Formatted at 2026-09-29 18:46:34 on dist-test-slave-smpn"
server_key: "873bd3cc52dbe1f54337cb861979c935"
server_key_iv: "ced6f4fbfb3473f139afc526c9626baf"
server_key_version: "encryptionkey@0"
I20260929 18:46:34.692211 29507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 18:46:34.694612 29522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:34.695864 29507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:46:34.695981 29507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "2eb6a1eabc8449949778a487700be416"
format_stamp: "Formatted at 2026-09-29 18:46:34 on dist-test-slave-smpn"
server_key: "873bd3cc52dbe1f54337cb861979c935"
server_key_iv: "ced6f4fbfb3473f139afc526c9626baf"
server_key_version: "encryptionkey@0"
I20260929 18:46:34.696077 29507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:34.725769 29507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:34.728647 29507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:34.728852 29507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:34.736012 29507 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:39935
I20260929 18:46:34.736040 29574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:39935 every 8 connection(s)
I20260929 18:46:34.736964 29507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:34.739660 29575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:34.744872 29575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416: Bootstrap starting.
I20260929 18:46:34.746089 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 29507
I20260929 18:46:34.746214 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:34.746479 7797 external_mini_cluster.cc:1428] Setting key ad11f9e678f1cbdf691de1ac3353e31f
I20260929 18:46:34.747694 29575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:34.748481 29575 log.cc:824] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:34.750342 29575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416: No bootstrap required, opened a new log
I20260929 18:46:34.754006 29575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb6a1eabc8449949778a487700be416" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39935 } }
I20260929 18:46:34.754230 29575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:34.754326 29575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb6a1eabc8449949778a487700be416, State: Initialized, Role: FOLLOWER
I20260929 18:46:34.754804 29575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb6a1eabc8449949778a487700be416" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39935 } }
I20260929 18:46:34.754933 29575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:34.754987 29575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:34.755115 29575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:34.755614 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:34.747625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56979 (local address 127.7.157.126:39935)
0929 18:46:34.748082 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:34.748097 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:34.748133 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:46:34.748860 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:34.748886 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:34.749318 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:34.749629 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:34.750040 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:34.751037 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:34.751991 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:34.752714 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:34.752870 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:34.753694 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:34.753700 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:34.753713 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:34.754114 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:34.754171 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:34.754194 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:34.754330 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:34.754528 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 18:46:34.754669 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":138,"threads_started":1}
I20260929 18:46:34.756052 29575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb6a1eabc8449949778a487700be416" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39935 } }
I20260929 18:46:34.756383 29575 leader_election.cc:302] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2eb6a1eabc8449949778a487700be416; no voters:
I20260929 18:46:34.757362 29575 leader_election.cc:288] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:34.757452 29580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:34.757653 29580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [term 1 LEADER]: Becoming Leader. State: Replica: 2eb6a1eabc8449949778a487700be416, State: Running, Role: LEADER
I20260929 18:46:34.757987 29580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb6a1eabc8449949778a487700be416" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39935 } }
I20260929 18:46:34.758424 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:34.759014 29575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:34.759609 29582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2eb6a1eabc8449949778a487700be416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb6a1eabc8449949778a487700be416" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39935 } } }
I20260929 18:46:34.759723 29582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:34.760025 29581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eb6a1eabc8449949778a487700be416. Latest consensus state: current_term: 1 leader_uuid: "2eb6a1eabc8449949778a487700be416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb6a1eabc8449949778a487700be416" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 39935 } } }
I20260929 18:46:34.760100 29581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:34.760944 29589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:34.763947 29589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:34.765599 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:34.760562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33535 (local address 127.7.157.126:39935)
0929 18:46:34.760669 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:34.760673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:34.761144 (+ 471us) server_negotiation.cc:407] Connection header received
0929 18:46:34.761304 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:34.761307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:34.761350 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:34.761398 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:34.761752 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:34.762495 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:34.763680 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:34.763988 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:34.764087 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:34.765109 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:34.765113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:34.765116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:34.765129 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:34.765159 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:34.765173 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:34.765264 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:34.765416 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 18:46:34.765453 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:46:34.769196 29589 catalog_manager.cc:1347] Generated new cluster ID: 679aef2cb73c4e958be7c6812701c275
I20260929 18:46:34.769261 29589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:34.779496 29589 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:34.780439 29589 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:34.788950 29589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2eb6a1eabc8449949778a487700be416: Generated new TSK 0
I20260929 18:46:34.789781 29589 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:34.801896 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 29507
W20260929 18:46:34.806523 29492 connection.cc:535] client connection to 127.7.157.126:39935 recv error: Network error: failed to read from TLS socket (remote: 127.7.157.126:39935): Connection reset by peer (error 104)
2026-09-29T18:46:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:34 dist-test-slave-smpn krb5kdc[29606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:34 dist-test-slave-smpn krb5kdc[29606](info): set up 2 sockets
Sep 29 18:46:34 dist-test-slave-smpn krb5kdc[29606](info): commencing operation
krb5kdc: starting...
W20260929 18:46:35.758952 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:35 dist-test-slave-smpn krb5kdc[29606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:46:35.873369 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:33549
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:33549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:35.979496 29619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:35.979768 29619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:35.979826 29619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:35.983779 29619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:35.983839 29619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:35.983865 29619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:35.983896 29619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:35.983917 29619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:35.989099 29619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:33549
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:33549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:35.990090 29619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:35.991106 29619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:46:35.998944 29619 server_base.cc:1034] running on GCE node
W20260929 18:46:35.998962 29627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:35.998953 29624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:35.999231 29625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:35.999753 29619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:36.000797 29619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:36.001956 29619 hybrid_clock.cc:648] HybridClock initialized: now 1790707596001942 us; error 25 us; skew 500 ppm
Sep 29 18:46:35 dist-test-slave-smpn krb5kdc[29606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707595, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:36.004576 29619 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:46:36.005450 29619 webserver.cc:460] Webserver started at http://127.7.157.126:44545/ using document root <none> and password file <none>
I20260929 18:46:36.005982 29619 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:36.006048 29619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:36.006244 29619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:36.008327 29619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "ac94f9854a39491aabd61e119b4b5c41"
format_stamp: "Formatted at 2026-09-29 18:46:35 on dist-test-slave-smpn"
server_key: "840be41b9ba92632d0a34a77d0691883"
server_key_iv: "510ed34dbf8dc6ef3c9af1b95bdccc10"
server_key_version: "encryptionkey@0"
I20260929 18:46:36.008863 29619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "ac94f9854a39491aabd61e119b4b5c41"
format_stamp: "Formatted at 2026-09-29 18:46:35 on dist-test-slave-smpn"
server_key: "840be41b9ba92632d0a34a77d0691883"
server_key_iv: "510ed34dbf8dc6ef3c9af1b95bdccc10"
server_key_version: "encryptionkey@0"
I20260929 18:46:36.012506 29619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:46:36.014786 29634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:36.015964 29619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:36.016075 29619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "ac94f9854a39491aabd61e119b4b5c41"
format_stamp: "Formatted at 2026-09-29 18:46:35 on dist-test-slave-smpn"
server_key: "840be41b9ba92632d0a34a77d0691883"
server_key_iv: "510ed34dbf8dc6ef3c9af1b95bdccc10"
server_key_version: "encryptionkey@0"
I20260929 18:46:36.016173 29619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:36.020699 29619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:36.023418 29619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:36.023582 29619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:36.030092 29619 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:33549
I20260929 18:46:36.030110 29686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:33549 every 8 connection(s)
I20260929 18:46:36.030972 29619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:36.033677 29687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:36.038383 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 29619
I20260929 18:46:36.038506 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:36.038755 7797 external_mini_cluster.cc:1428] Setting key ae21ce31b1830c18fa89605dfa4332a9
I20260929 18:46:36.039243 29687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41: Bootstrap starting.
I20260929 18:46:36.041625 29687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:36.042474 29687 log.cc:824] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:36.043830 29690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:36.040073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49789 (local address 127.7.157.126:33549)
0929 18:46:36.040493 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:36.040508 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:36.040544 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:46:36.041178 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:36.041194 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:36.041487 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:36.041809 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:36.042578 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:36.042584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:36.042601 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:36.042612 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:36.042665 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:36.042686 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:36.042784 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:36.042920 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 18:46:36.043097 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":130,"threads_started":1}
I20260929 18:46:36.044430 29687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41: No bootstrap required, opened a new log
I20260929 18:46:36.046582 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:36.048758 29687 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac94f9854a39491aabd61e119b4b5c41" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33549 } }
I20260929 18:46:36.049031 29687 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:36.049144 29687 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac94f9854a39491aabd61e119b4b5c41, State: Initialized, Role: FOLLOWER
I20260929 18:46:36.049654 29687 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac94f9854a39491aabd61e119b4b5c41" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33549 } }
I20260929 18:46:36.049822 29687 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:36.049898 29687 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:36.050000 29687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:36.050299 29690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:36.048845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39289 (local address 127.7.157.126:33549)
0929 18:46:36.048954 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:36.048959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:36.049087 (+ 128us) server_negotiation.cc:407] Connection header received
0929 18:46:36.049213 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:36.049215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:36.049246 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:36.049320 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:36.049905 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:36.049908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:36.049911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:36.049922 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:36.049949 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:36.049963 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:36.050040 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:36.050148 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 18:46:36.050184 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:46:36.051225 29687 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac94f9854a39491aabd61e119b4b5c41" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33549 } }
I20260929 18:46:36.051545 29687 leader_election.cc:302] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac94f9854a39491aabd61e119b4b5c41; no voters:
I20260929 18:46:36.051813 29687 leader_election.cc:288] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:36.051895 29697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:36.052076 29697 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [term 1 LEADER]: Becoming Leader. State: Replica: ac94f9854a39491aabd61e119b4b5c41, State: Running, Role: LEADER
I20260929 18:46:36.052438 29697 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac94f9854a39491aabd61e119b4b5c41" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33549 } }
I20260929 18:46:36.053009 29687 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:36.054047 29698 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac94f9854a39491aabd61e119b4b5c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac94f9854a39491aabd61e119b4b5c41" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33549 } } }
I20260929 18:46:36.054189 29698 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:36.054055 29699 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac94f9854a39491aabd61e119b4b5c41. Latest consensus state: current_term: 1 leader_uuid: "ac94f9854a39491aabd61e119b4b5c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac94f9854a39491aabd61e119b4b5c41" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 33549 } } }
I20260929 18:46:36.054247 29699 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:36.054594 29706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:36.057878 29706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:36.064241 29706 catalog_manager.cc:1347] Generated new cluster ID: 50d5fd96811b4ec2afd74efb61c301af
I20260929 18:46:36.064311 29706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:36.084295 29706 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:36.084717 29706 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:36.094929 29706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac94f9854a39491aabd61e119b4b5c41: Generated new TSK 0
I20260929 18:46:36.095717 29706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:36.102383 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 29619
2026-09-29T18:46:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:36 dist-test-slave-smpn krb5kdc[29718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:36 dist-test-slave-smpn krb5kdc[29718](info): set up 2 sockets
Sep 29 18:46:36 dist-test-slave-smpn krb5kdc[29718](info): commencing operation
krb5kdc: starting...
W20260929 18:46:37.042994 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:37 dist-test-slave-smpn krb5kdc[29718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:46:37.155701 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:45727
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:45727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:37.263383 29731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:37.263659 29731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:37.263718 29731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:37.267685 29731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:37.267752 29731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:37.267777 29731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:37.267799 29731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:37.267827 29731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:37.273128 29731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:45727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:45727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:37.274160 29731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:37.275185 29731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:37.283298 29737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:37.283317 29739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:37.283310 29736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:37.283365 29731 server_base.cc:1034] running on GCE node
I20260929 18:46:37.283931 29731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:37.285004 29731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:37.286176 29731 hybrid_clock.cc:648] HybridClock initialized: now 1790707597286153 us; error 36 us; skew 500 ppm
Sep 29 18:46:37 dist-test-slave-smpn krb5kdc[29718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707597, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:37.293390 29731 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:46:37.294363 29731 webserver.cc:460] Webserver started at http://127.7.157.126:36513/ using document root <none> and password file <none>
I20260929 18:46:37.294930 29731 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:37.294999 29731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:37.295251 29731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:37.297326 29731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "3f9ee7ab9c054f62b7c3be19429b594e"
format_stamp: "Formatted at 2026-09-29 18:46:37 on dist-test-slave-smpn"
server_key: "33e80f233e7ba6158a55d4aeaad5ec24"
server_key_iv: "2b2782e4de082efc1897aa11c7cefc3f"
server_key_version: "encryptionkey@0"
I20260929 18:46:37.297864 29731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "3f9ee7ab9c054f62b7c3be19429b594e"
format_stamp: "Formatted at 2026-09-29 18:46:37 on dist-test-slave-smpn"
server_key: "33e80f233e7ba6158a55d4aeaad5ec24"
server_key_iv: "2b2782e4de082efc1897aa11c7cefc3f"
server_key_version: "encryptionkey@0"
I20260929 18:46:37.301615 29731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:46:37.303906 29746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:37.305070 29731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:37.305186 29731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "3f9ee7ab9c054f62b7c3be19429b594e"
format_stamp: "Formatted at 2026-09-29 18:46:37 on dist-test-slave-smpn"
server_key: "33e80f233e7ba6158a55d4aeaad5ec24"
server_key_iv: "2b2782e4de082efc1897aa11c7cefc3f"
server_key_version: "encryptionkey@0"
I20260929 18:46:37.305291 29731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:37.320437 29731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:37.323165 29731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:37.323359 29731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:37.329903 29731 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:45727
I20260929 18:46:37.329916 29798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:45727 every 8 connection(s)
I20260929 18:46:37.330828 29731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:37.331230 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 29731
I20260929 18:46:37.331346 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:37.331622 7797 external_mini_cluster.cc:1428] Setting key 19c2250914518c3fa07ffe8480ffc60e
I20260929 18:46:37.333848 29799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:37.339279 29799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e: Bootstrap starting.
Sep 29 18:46:37 dist-test-slave-smpn krb5kdc[29718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:37.341933 29799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:37.342634 29799 log.cc:824] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:37.344848 29799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e: No bootstrap required, opened a new log
I20260929 18:46:37.345391 29802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:37.332936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38575 (local address 127.7.157.126:45727)
0929 18:46:37.333352 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:37.333366 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:37.333400 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:46:37.334049 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:37.334067 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:37.334395 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:37.334703 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:37.335489 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:37.336643 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:37.337486 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:37.337950 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:37.338096 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:37.340897 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:37.340902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:37.340914 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:37.340941 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:37.342604 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:37.343087 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:37.343093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:37.343099 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:37.343240 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:37.343485 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:37.343488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:37.343489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:37.343904 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:37.344047 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:37.344331 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 18:46:37.344601 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":138,"threads_started":1}
I20260929 18:46:37.348094 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:37.348448 29799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9ee7ab9c054f62b7c3be19429b594e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45727 } }
I20260929 18:46:37.348660 29799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:37.348726 29799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9ee7ab9c054f62b7c3be19429b594e, State: Initialized, Role: FOLLOWER
I20260929 18:46:37.349117 29799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9ee7ab9c054f62b7c3be19429b594e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45727 } }
I20260929 18:46:37.349331 29799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:37.349408 29799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:37.349509 29799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:37.350435 29799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9ee7ab9c054f62b7c3be19429b594e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45727 } }
I20260929 18:46:37.350807 29799 leader_election.cc:302] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f9ee7ab9c054f62b7c3be19429b594e; no voters:
I20260929 18:46:37.351166 29799 leader_election.cc:288] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:37.351248 29809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:37.351462 29809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [term 1 LEADER]: Becoming Leader. State: Replica: 3f9ee7ab9c054f62b7c3be19429b594e, State: Running, Role: LEADER
I20260929 18:46:37.351857 29809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9ee7ab9c054f62b7c3be19429b594e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45727 } }
I20260929 18:46:37.352366 29799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:37.353741 29811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f9ee7ab9c054f62b7c3be19429b594e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9ee7ab9c054f62b7c3be19429b594e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45727 } } }
I20260929 18:46:37.353897 29811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:37.353869 29810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f9ee7ab9c054f62b7c3be19429b594e. Latest consensus state: current_term: 1 leader_uuid: "3f9ee7ab9c054f62b7c3be19429b594e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9ee7ab9c054f62b7c3be19429b594e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45727 } } }
I20260929 18:46:37.353996 29810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:37.354821 29821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:37.357673 29821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:37.358785 29802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:37.350059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48119 (local address 127.7.157.126:45727)
0929 18:46:37.350177 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:37.350182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:37.350385 (+ 203us) server_negotiation.cc:407] Connection header received
0929 18:46:37.350483 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:37.350486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:37.350529 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:37.350595 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:37.351434 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:37.352160 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:37.353271 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:37.353584 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:37.353678 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:37.355243 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:37.355247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:37.355249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:37.355274 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:37.356743 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:37.357205 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:37.357209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:37.357211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:37.357296 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:37.357572 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:37.357576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:37.357577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:37.357800 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:37.357913 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:37.358514 (+ 601us) server_negotiation.cc:299] Negotiation successful
0929 18:46:37.358626 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:46:37.362607 29821 catalog_manager.cc:1347] Generated new cluster ID: e3f402afd28c466aa083e5e114c6fcde
I20260929 18:46:37.362666 29821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:37.372418 29821 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:37.373615 29821 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:37.381274 29821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f9ee7ab9c054f62b7c3be19429b594e: Generated new TSK 0
I20260929 18:46:37.382030 29821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:37.389560 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 29731
2026-09-29T18:46:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:37 dist-test-slave-smpn krb5kdc[29830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:37 dist-test-slave-smpn krb5kdc[29830](info): set up 2 sockets
Sep 29 18:46:37 dist-test-slave-smpn krb5kdc[29830](info): commencing operation
krb5kdc: starting...
W20260929 18:46:38.343021 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:38 dist-test-slave-smpn krb5kdc[29830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:46:38.457067 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:37647
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:37647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:38.562264 29843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:38.562498 29843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:38.562536 29843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:38.566347 29843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:38.566398 29843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:38.566416 29843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:38.566429 29843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:38.566443 29843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:38.571698 29843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:37647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:37647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:38.572647 29843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:38.573591 29843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:38.581669 29848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:38.581804 29843 server_base.cc:1034] running on GCE node
W20260929 18:46:38.581669 29849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:38.581669 29851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:38.582479 29843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:38.583582 29843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:38.584731 29843 hybrid_clock.cc:648] HybridClock initialized: now 1790707598584708 us; error 36 us; skew 500 ppm
Sep 29 18:46:38 dist-test-slave-smpn krb5kdc[29830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707598, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:38.587256 29843 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:46:38.588104 29843 webserver.cc:460] Webserver started at http://127.7.157.126:34821/ using document root <none> and password file <none>
I20260929 18:46:38.588608 29843 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:38.588672 29843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:38.588856 29843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:38.590780 29843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "2681c93453cb4479abd852c467e5287e"
format_stamp: "Formatted at 2026-09-29 18:46:38 on dist-test-slave-smpn"
server_key: "58edc1a446038c428891c67552992fdb"
server_key_iv: "e74408e3a9eddc0ec90c43d2dc91cb8c"
server_key_version: "encryptionkey@0"
I20260929 18:46:38.591298 29843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "2681c93453cb4479abd852c467e5287e"
format_stamp: "Formatted at 2026-09-29 18:46:38 on dist-test-slave-smpn"
server_key: "58edc1a446038c428891c67552992fdb"
server_key_iv: "e74408e3a9eddc0ec90c43d2dc91cb8c"
server_key_version: "encryptionkey@0"
I20260929 18:46:38.594662 29843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 18:46:38.596860 29858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:38.597975 29843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:46:38.598088 29843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "2681c93453cb4479abd852c467e5287e"
format_stamp: "Formatted at 2026-09-29 18:46:38 on dist-test-slave-smpn"
server_key: "58edc1a446038c428891c67552992fdb"
server_key_iv: "e74408e3a9eddc0ec90c43d2dc91cb8c"
server_key_version: "encryptionkey@0"
I20260929 18:46:38.598172 29843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:38.609459 29843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:38.612267 29843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:38.612424 29843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:38.618810 29843 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:37647
I20260929 18:46:38.618839 29910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:37647 every 8 connection(s)
I20260929 18:46:38.619704 29843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:38.622357 29911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:38.622702 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 29843
I20260929 18:46:38.622812 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:38.623123 7797 external_mini_cluster.cc:1428] Setting key 72c7eb8e6c29a668a2bbec5f78b305f1
I20260929 18:46:38.628962 29911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e: Bootstrap starting.
I20260929 18:46:38.631175 29911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:38.631479 29914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:38.624362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45493 (local address 127.7.157.126:37647)
0929 18:46:38.624736 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:38.624751 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:38.624784 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:46:38.625433 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:38.625450 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:38.625768 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:38.626042 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:38.626412 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:38.627417 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:38.628599 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:38.629041 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:38.629177 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:38.629906 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:38.629910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:38.629922 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:38.630278 (+ 356us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:38.630326 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:38.630347 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:38.630427 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:38.630547 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 18:46:38.630742 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":103,"threads_started":1}
I20260929 18:46:38.631994 29911 log.cc:824] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:38.633880 29911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e: No bootstrap required, opened a new log
I20260929 18:46:38.634239 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:38.637463 29911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2681c93453cb4479abd852c467e5287e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37647 } }
I20260929 18:46:38.637730 29911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:38.637804 29911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2681c93453cb4479abd852c467e5287e, State: Initialized, Role: FOLLOWER
I20260929 18:46:38.638253 29911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2681c93453cb4479abd852c467e5287e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37647 } }
I20260929 18:46:38.638410 29911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:38.638479 29911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:38.638571 29911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:38.639699 29911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2681c93453cb4479abd852c467e5287e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37647 } }
I20260929 18:46:38.640017 29914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:38.636128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58317 (local address 127.7.157.126:37647)
0929 18:46:38.636230 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:38.636234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:38.636374 (+ 140us) server_negotiation.cc:407] Connection header received
0929 18:46:38.636479 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:38.636481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:38.636549 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:38.636610 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:38.636861 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:38.637335 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:38.638379 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:38.638786 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:38.638882 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:38.639622 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:38.639625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:38.639627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:38.639637 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:38.639660 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:38.639670 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:38.639720 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:38.639831 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 18:46:38.639882 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:46:38.640055 29911 leader_election.cc:302] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2681c93453cb4479abd852c467e5287e; no voters:
I20260929 18:46:38.640394 29911 leader_election.cc:288] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:38.640463 29921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:38.641134 29921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [term 1 LEADER]: Becoming Leader. State: Replica: 2681c93453cb4479abd852c467e5287e, State: Running, Role: LEADER
I20260929 18:46:38.641456 29921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2681c93453cb4479abd852c467e5287e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37647 } }
I20260929 18:46:38.641537 29911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:38.642799 29923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2681c93453cb4479abd852c467e5287e. Latest consensus state: current_term: 1 leader_uuid: "2681c93453cb4479abd852c467e5287e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2681c93453cb4479abd852c467e5287e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37647 } } }
I20260929 18:46:38.642905 29923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:38.643224 29922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2681c93453cb4479abd852c467e5287e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2681c93453cb4479abd852c467e5287e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 37647 } } }
I20260929 18:46:38.643299 29922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e [sys.catalog]: This master's current role is: LEADER
W20260929 18:46:38.645840 29936 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:46:38.645905 29936 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:46:38.646005 29933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:38.646878 29933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:38.652576 29933 catalog_manager.cc:1347] Generated new cluster ID: d7e57a04179f4ed78c6b3c00022920ef
I20260929 18:46:38.652635 29933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:38.667999 29933 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:38.668941 29933 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:38.673496 29933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2681c93453cb4479abd852c467e5287e: Generated new TSK 0
I20260929 18:46:38.674173 29933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:38.681399 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 29843
2026-09-29T18:46:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:38 dist-test-slave-smpn krb5kdc[29942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:38 dist-test-slave-smpn krb5kdc[29942](info): set up 2 sockets
Sep 29 18:46:38 dist-test-slave-smpn krb5kdc[29942](info): commencing operation
krb5kdc: starting...
W20260929 18:46:39.623270 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:39 dist-test-slave-smpn krb5kdc[29942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:46:39.738605 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:44993
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:44993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:39.845772 29955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:39.846047 29955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:39.846107 29955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:39.850072 29955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:39.850128 29955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:39.850154 29955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:39.850175 29955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:39.850205 29955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:39.856215 29955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:44993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:44993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:39.857234 29955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:39.858193 29955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:39.866142 29961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:39.866142 29960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:39.866150 29963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:39.866464 29955 server_base.cc:1034] running on GCE node
I20260929 18:46:39.866961 29955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:39.868014 29955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:39.869169 29955 hybrid_clock.cc:648] HybridClock initialized: now 1790707599869153 us; error 26 us; skew 500 ppm
Sep 29 18:46:39 dist-test-slave-smpn krb5kdc[29942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707599, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:39.871727 29955 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:46:39.872555 29955 webserver.cc:460] Webserver started at http://127.7.157.126:33437/ using document root <none> and password file <none>
I20260929 18:46:39.873055 29955 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:39.873114 29955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:39.873296 29955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:39.875272 29955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "2ebe5494c1d14091b7c81290ff5b1284"
format_stamp: "Formatted at 2026-09-29 18:46:39 on dist-test-slave-smpn"
server_key: "69105ed7f7b3d2167d97ff795cbff5fd"
server_key_iv: "5b40089823f8e6f3a9629dcb8cc1abb1"
server_key_version: "encryptionkey@0"
I20260929 18:46:39.875782 29955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "2ebe5494c1d14091b7c81290ff5b1284"
format_stamp: "Formatted at 2026-09-29 18:46:39 on dist-test-slave-smpn"
server_key: "69105ed7f7b3d2167d97ff795cbff5fd"
server_key_iv: "5b40089823f8e6f3a9629dcb8cc1abb1"
server_key_version: "encryptionkey@0"
I20260929 18:46:39.879328 29955 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:46:39.881753 29970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:39.882993 29955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:46:39.883149 29955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "2ebe5494c1d14091b7c81290ff5b1284"
format_stamp: "Formatted at 2026-09-29 18:46:39 on dist-test-slave-smpn"
server_key: "69105ed7f7b3d2167d97ff795cbff5fd"
server_key_iv: "5b40089823f8e6f3a9629dcb8cc1abb1"
server_key_version: "encryptionkey@0"
I20260929 18:46:39.883265 29955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:39.887897 29955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:39.890841 29955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:39.890998 29955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:39.897505 29955 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:44993
I20260929 18:46:39.897545 30022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:44993 every 8 connection(s)
I20260929 18:46:39.898392 29955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:39.901084 30023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:39.904045 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 29955
I20260929 18:46:39.904160 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:39.904395 7797 external_mini_cluster.cc:1428] Setting key 433a74fddd99f83c57bdd5537695dfd7
I20260929 18:46:39.906849 30023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284: Bootstrap starting.
I20260929 18:46:39.909685 30023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:39.909893 30026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:39.905737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42353 (local address 127.7.157.126:44993)
0929 18:46:39.906163 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:39.906182 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:39.906223 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:46:39.907013 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:39.907035 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:39.907410 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:39.907761 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:39.908394 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:39.908401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:39.908415 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:39.908427 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:39.908488 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:39.908524 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:39.908662 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:39.908798 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 18:46:39.908953 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":110,"threads_started":1}
I20260929 18:46:39.910497 30023 log.cc:824] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:39.912581 30023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284: No bootstrap required, opened a new log
I20260929 18:46:39.912716 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:39.916620 30026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:39.914605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35871 (local address 127.7.157.126:44993)
0929 18:46:39.914701 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:39.914704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:39.915353 (+ 649us) server_negotiation.cc:407] Connection header received
0929 18:46:39.915448 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:39.915450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:39.915487 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:39.915584 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:39.916239 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:39.916242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:39.916244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:46:39.916253 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:39.916279 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:46:39.916289 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:39.916354 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:39.916440 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 18:46:39.916471 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:46:39.916622 30023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebe5494c1d14091b7c81290ff5b1284" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44993 } }
I20260929 18:46:39.916963 30023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:46:39.917037 30023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebe5494c1d14091b7c81290ff5b1284, State: Initialized, Role: FOLLOWER
I20260929 18:46:39.917443 30023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebe5494c1d14091b7c81290ff5b1284" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44993 } }
I20260929 18:46:39.917608 30023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:46:39.917671 30023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:46:39.917768 30023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:39.918618 30023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebe5494c1d14091b7c81290ff5b1284" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44993 } }
I20260929 18:46:39.918922 30023 leader_election.cc:302] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ebe5494c1d14091b7c81290ff5b1284; no voters:
I20260929 18:46:39.919220 30023 leader_election.cc:288] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:46:39.919306 30033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:39.919510 30033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [term 1 LEADER]: Becoming Leader. State: Replica: 2ebe5494c1d14091b7c81290ff5b1284, State: Running, Role: LEADER
I20260929 18:46:39.919833 30033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebe5494c1d14091b7c81290ff5b1284" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44993 } }
I20260929 18:46:39.920354 30023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:39.921373 30035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ebe5494c1d14091b7c81290ff5b1284. Latest consensus state: current_term: 1 leader_uuid: "2ebe5494c1d14091b7c81290ff5b1284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebe5494c1d14091b7c81290ff5b1284" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44993 } } }
I20260929 18:46:39.921495 30035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:39.921769 30034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ebe5494c1d14091b7c81290ff5b1284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebe5494c1d14091b7c81290ff5b1284" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 44993 } } }
I20260929 18:46:39.921852 30034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284 [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:39.921904 30042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:39.924849 30042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:39.930253 30042 catalog_manager.cc:1347] Generated new cluster ID: 9eacbc3876a54e01b0cd0da014634db0
I20260929 18:46:39.930325 30042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:39.942695 30042 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:39.943034 30042 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:39.954301 30042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ebe5494c1d14091b7c81290ff5b1284: Generated new TSK 0
I20260929 18:46:39.955183 30042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:46:39.960093 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 29955
2026-09-29T18:46:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1278 ms)
[----------] 18 tests from AuthTokenIssuingTest (23468 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:40 dist-test-slave-smpn krb5kdc[30054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:40 dist-test-slave-smpn krb5kdc[30054](info): set up 2 sockets
Sep 29 18:46:40 dist-test-slave-smpn krb5kdc[30054](info): commencing operation
krb5kdc: starting...
W20260929 18:46:40.911046 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:40 dist-test-slave-smpn krb5kdc[30054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:46:41.023375 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:45793
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:38335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:45793,127.7.157.125:43667,127.7.157.124:37257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:41.128079 30067 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:46:41.128393 30067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:41.128434 30067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:41.128499 30067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:41.132555 30067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:41.132619 30067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:41.132647 30067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:41.132668 30067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:41.132688 30067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:41.138055 30067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:38335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:45793,127.7.157.125:43667,127.7.157.124:37257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:45793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:41.139151 30067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:41.140179 30067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:41.148265 30072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:41.148253 30073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:41.148253 30075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:41.148459 30067 server_base.cc:1034] running on GCE node
I20260929 18:46:41.149034 30067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:41.150106 30067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:41.151259 30067 hybrid_clock.cc:648] HybridClock initialized: now 1790707601151233 us; error 37 us; skew 500 ppm
Sep 29 18:46:41 dist-test-slave-smpn krb5kdc[30054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707601, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:41.153877 30067 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:46:41.154729 30067 webserver.cc:460] Webserver started at http://127.7.157.126:42639/ using document root <none> and password file <none>
I20260929 18:46:41.155292 30067 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:41.155359 30067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:41.155589 30067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:41.157541 30067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec"
format_stamp: "Formatted at 2026-09-29 18:46:41 on dist-test-slave-smpn"
server_key: "eddb284cb426c4173e5a00b13025a051"
server_key_iv: "c144c8c62c41482c1874e6d115a230cc"
server_key_version: "encryptionkey@0"
I20260929 18:46:41.158054 30067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec"
format_stamp: "Formatted at 2026-09-29 18:46:41 on dist-test-slave-smpn"
server_key: "eddb284cb426c4173e5a00b13025a051"
server_key_iv: "c144c8c62c41482c1874e6d115a230cc"
server_key_version: "encryptionkey@0"
I20260929 18:46:41.161584 30067 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 18:46:41.163859 30082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:41.165083 30067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 18:46:41.165200 30067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec"
format_stamp: "Formatted at 2026-09-29 18:46:41 on dist-test-slave-smpn"
server_key: "eddb284cb426c4173e5a00b13025a051"
server_key_iv: "c144c8c62c41482c1874e6d115a230cc"
server_key_version: "encryptionkey@0"
I20260929 18:46:41.165297 30067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:41.199968 30067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:41.202769 30067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:41.202946 30067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:41.209478 30067 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:45793
I20260929 18:46:41.209736 30134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:45793 every 8 connection(s)
I20260929 18:46:41.210456 30067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:41.213218 30135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:41.217175 30135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:46:41.219175 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 30067
I20260929 18:46:41.219280 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:41.219522 7797 external_mini_cluster.cc:1428] Setting key c7f102669e0cee3d14702a9b1a0f8a7b
Sep 29 18:46:41 dist-test-slave-smpn krb5kdc[30054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707601, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.157.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.125@KRBTEST.COM" created.
I20260929 18:46:41.232705 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.220289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.7.157.126:45793)
0929 18:46:41.220559 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.220571 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.220611 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:46:41.221519 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.221525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.221768 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.221963 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.222834 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.223803 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.225221 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.225615 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.225726 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.228220 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:41.228226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:41.228235 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:41.228256 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:41.229769 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.230207 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.230213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.230214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.230318 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.230623 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.230626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.230628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.230987 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:41.231125 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:41.231457 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 18:46:41.231676 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":83,"threads_started":1}
I20260929 18:46:41.232705 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.219457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45793 (local address 127.0.0.1:55324)
0929 18:46:41.220025 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.220085 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:41.221131 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:41.221991 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:41.221996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:41.222327 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:41.222658 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.222694 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.223923 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.223926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.225100 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.225112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.225774 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.225778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.225897 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.226437 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:41.226459 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:41.228090 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:41.229848 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.229854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.229862 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.230111 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.230400 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.230403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.230405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.230544 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.231106 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:41.231112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:41.231238 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 18:46:41.231381 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 18:46:41.231756 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":93,"thread_start_us":130,"threads_started":1}
I20260929 18:46:41.235033 30135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:46:41.235781 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.235544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:43667 (local address 127.0.0.1:55148)
0929 18:46:41.235651 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.235720 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 18:46:41.236006 30083 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.157.125:43667: connect: Connection refused (error 111)
I20260929 18:46:41.236819 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.236628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:43667 (local address 127.0.0.1:55158)
0929 18:46:41.236727 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.236769 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 18:46:41.236954 30135 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.125:43667: Network error: Client connection negotiation failed: client connection to 127.7.157.125:43667: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.125.keytab.
Entry for principal kudu/127.7.157.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.125.keytab.
I20260929 18:46:41.244156 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.125:43667
--webserver_interface=127.7.157.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.125.keytab
--principal=kudu/127.7.157.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:38335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:45793,127.7.157.125:43667,127.7.157.124:37257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:46:41.286206 30135 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } attempt: 1
I20260929 18:46:41.287041 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.286759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:43667 (local address 127.0.0.1:55160)
0929 18:46:41.286896 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.286957 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 18:46:41.287786 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.287591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:43667 (local address 127.0.0.1:55172)
0929 18:46:41.287689 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.287733 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 18:46:41.287915 30135 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.125:43667: Network error: Client connection negotiation failed: client connection to 127.7.157.125:43667: connect: Connection refused (error 111)
W20260929 18:46:41.351799 30142 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:46:41.352072 30142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:41.352094 30142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:41.352129 30142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:41.356032 30142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:41.356086 30142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:41.356103 30142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:41.356119 30142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:41.356132 30142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:41.356067 30135 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } attempt: 2
I20260929 18:46:41.356868 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.356590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:43667 (local address 127.0.0.1:55180)
0929 18:46:41.356730 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.356792 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 18:46:41.357540 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.357337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:43667 (local address 127.0.0.1:55194)
0929 18:46:41.357432 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.357475 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 18:46:41.357647 30135 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.125:43667: Network error: Client connection negotiation failed: client connection to 127.7.157.125:43667: connect: Connection refused (error 111)
I20260929 18:46:41.361617 30142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:38335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:45793,127.7.157.125:43667,127.7.157.124:37257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.125.keytab
--principal=kudu/127.7.157.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.125:43667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.157.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:41.362542 30142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:41.363524 30142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:41.371531 30150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:41.371541 30148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:41.371634 30142 server_base.cc:1034] running on GCE node
W20260929 18:46:41.371531 30147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:41.372313 30142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:41.373406 30142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:41.374562 30142 hybrid_clock.cc:648] HybridClock initialized: now 1790707601374541 us; error 36 us; skew 500 ppm
Sep 29 18:46:41 dist-test-slave-smpn krb5kdc[30054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707601, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:41.377254 30142 init.cc:377] Logged in from keytab as kudu/127.7.157.125@KRBTEST.COM (short username kudu)
I20260929 18:46:41.378098 30142 webserver.cc:460] Webserver started at http://127.7.157.125:34615/ using document root <none> and password file <none>
I20260929 18:46:41.378633 30142 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:41.378700 30142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:41.378894 30142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:41.381040 30142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/data/instance:
uuid: "41038e49760c490d96dbea59e4f13d4c"
format_stamp: "Formatted at 2026-09-29 18:46:41 on dist-test-slave-smpn"
server_key: "d5de9285839347e62399e86a9a14ad5e"
server_key_iv: "d01c64c5ba5390a86f3889bcb613447e"
server_key_version: "encryptionkey@0"
I20260929 18:46:41.381592 30142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/wal/instance:
uuid: "41038e49760c490d96dbea59e4f13d4c"
format_stamp: "Formatted at 2026-09-29 18:46:41 on dist-test-slave-smpn"
server_key: "d5de9285839347e62399e86a9a14ad5e"
server_key_iv: "d01c64c5ba5390a86f3889bcb613447e"
server_key_version: "encryptionkey@0"
I20260929 18:46:41.385247 30142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:46:41.387579 30157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:41.388738 30142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:41.388844 30142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/wal
uuid: "41038e49760c490d96dbea59e4f13d4c"
format_stamp: "Formatted at 2026-09-29 18:46:41 on dist-test-slave-smpn"
server_key: "d5de9285839347e62399e86a9a14ad5e"
server_key_iv: "d01c64c5ba5390a86f3889bcb613447e"
server_key_version: "encryptionkey@0"
I20260929 18:46:41.388932 30142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:41.405592 30142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:41.408411 30142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:41.408571 30142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:41.415105 30142 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.125:43667
I20260929 18:46:41.415337 30209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.125:43667 every 8 connection(s)
I20260929 18:46:41.416007 30142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/data/info.pb
I20260929 18:46:41.418565 30210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:41.420029 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 30142
I20260929 18:46:41.420121 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-1/wal/instance
I20260929 18:46:41.420384 7797 external_mini_cluster.cc:1428] Setting key fff4b8afa9b96dcc09b3c240b03e8774
I20260929 18:46:41.422251 30210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.157.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.124@KRBTEST.COM" created.
Sep 29 18:46:41 dist-test-slave-smpn krb5kdc[30054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707601, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.125@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:41.435802 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.424045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.7.157.126:45793)
0929 18:46:41.424215 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.424221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.425042 (+ 821us) server_negotiation.cc:407] Connection header received
0929 18:46:41.426275 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.426278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.426330 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.426406 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.427977 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.428543 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.429683 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.429955 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.430049 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.432431 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:41.432434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:41.432436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:41.432463 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:41.433770 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.434265 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.434269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.434270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.434347 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.434660 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.434663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.434665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.434845 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:41.434927 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:41.435494 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 18:46:41.435634 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:46:41.436618 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.424357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45793 (local address 127.0.0.1:55334)
0929 18:46:41.424857 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.424918 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:41.426006 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:41.426609 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:41.426618 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:41.427061 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:41.427788 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.427810 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.428664 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.428668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.429577 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.429586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.430095 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.430098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.430261 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.430807 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:41.430824 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:41.432259 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:41.433881 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.433887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.433904 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.434174 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.434428 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.434432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.434434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.434577 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.434941 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:41.434947 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:41.435254 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 18:46:41.435430 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 18:46:41.435767 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":111,"threads_started":1}
I20260929 18:46:41.438024 30210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:46:41 dist-test-slave-smpn krb5kdc[30054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707601, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.125@KRBTEST.COM for kudu/127.7.157.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.124.keytab.
Entry for principal kudu/127.7.157.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.124.keytab.
I20260929 18:46:41.445535 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.124:37257
--webserver_interface=127.7.157.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.124.keytab
--principal=kudu/127.7.157.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:38335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:45793,127.7.157.125:43667,127.7.157.124:37257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:46:41.446713 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.438507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:43667 (local address 127.0.0.1:55198)
0929 18:46:41.438619 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.438631 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:41.438711 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:41.439308 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:41.439310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:41.439454 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:41.439551 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.439556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.440535 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.440537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.441111 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.441120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.441777 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.441779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.441852 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.442274 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:41.442286 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:41.443536 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:41.445348 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.445350 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.445351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.445529 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.445810 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.445813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.445814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.445889 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.446342 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:41.446344 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:41.446416 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 18:46:41.446498 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:46:41.446596 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 18:46:41.446903 30215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.438639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.7.157.125:43667)
0929 18:46:41.438918 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.438929 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.438951 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:46:41.438989 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.438993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.439200 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.439304 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.439658 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.440445 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.441229 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.441675 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.441758 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.443650 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:41.443659 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:41.443672 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:41.443694 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:41.445251 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.445633 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.445640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.445642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.445731 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.445958 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.445961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.445962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.446252 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:41.446373 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:41.446587 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 18:46:41.446737 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":100,"threads_started":1}
I20260929 18:46:41.448817 30135 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } attempt: 3
I20260929 18:46:41.448973 30210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:46:41.449841 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.449595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:37257 (local address 127.0.0.1:45380)
0929 18:46:41.449697 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.449774 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 18:46:41.450024 30160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
I20260929 18:46:41.450882 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.450658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:37257 (local address 127.0.0.1:45392)
0929 18:46:41.450765 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.450818 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:46:41.451059 30210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:37257: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
Sep 29 18:46:41 dist-test-slave-smpn krb5kdc[30054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707601, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for kudu/127.7.157.125@KRBTEST.COM
I20260929 18:46:41.458392 30215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.449728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.7.157.125:43667)
0929 18:46:41.449850 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.449855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.449869 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:46:41.449903 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.449906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.449949 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.450028 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.450550 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.451280 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.452889 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.453128 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.453202 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.455401 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:41.455404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:41.455406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:41.455428 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:41.456940 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.457392 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.457394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.457396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.457474 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.457706 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.457708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.457709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.457866 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:41.457940 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:41.458187 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 18:46:41.458279 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:46:41.458447 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.449273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:43667 (local address 127.0.0.1:55200)
0929 18:46:41.449382 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.449397 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:41.449475 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:41.450100 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:41.450104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:41.450289 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:41.450421 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.450428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.451398 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.451402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.452751 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.452765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.453239 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.453243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.453357 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.453774 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:41.453789 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:41.455248 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:41.457052 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.457056 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.457058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.457306 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.457546 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.457549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.457551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.457634 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.457940 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:41.457943 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:41.458044 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 18:46:41.458146 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 18:46:41.458285 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 18:46:41.459517 30135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:46:41.460242 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.459968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:37257 (local address 127.0.0.1:45394)
0929 18:46:41.460090 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.460149 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 18:46:41.460949 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.460704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:37257 (local address 127.0.0.1:45404)
0929 18:46:41.460839 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.460887 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260929 18:46:41.461184 30135 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:37257: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
I20260929 18:46:41.492352 30135 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } attempt: 1
I20260929 18:46:41.493180 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.492875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:37257 (local address 127.0.0.1:45412)
0929 18:46:41.493017 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.493082 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 18:46:41.493852 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.493646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:37257 (local address 127.0.0.1:45414)
0929 18:46:41.493754 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.493798 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 18:46:41.493975 30135 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:37257: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
I20260929 18:46:41.500345 30210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } attempt: 1
I20260929 18:46:41.501143 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.500849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:37257 (local address 127.0.0.1:45418)
0929 18:46:41.500993 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.501059 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 18:46:41.501847 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.501639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:37257 (local address 127.0.0.1:45420)
0929 18:46:41.501734 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.501783 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 18:46:41.501967 30210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:37257: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
W20260929 18:46:41.555907 30216 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:46:41.556202 30216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:41.556226 30216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:41.556275 30216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:41.560297 30216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:41.560360 30216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:41.560379 30216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:41.560393 30216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:41.560406 30216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:41.565680 30216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:38335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:45793,127.7.157.125:43667,127.7.157.124:37257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.124.keytab
--principal=kudu/127.7.157.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.124:37257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.157.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:41.566663 30216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:41.567721 30216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:46:41.569114 30135 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } attempt: 2
I20260929 18:46:41.569852 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.569593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:37257 (local address 127.0.0.1:45428)
0929 18:46:41.569725 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.569781 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 18:46:41.570111 30210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } attempt: 2
I20260929 18:46:41.570497 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.570295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:37257 (local address 127.0.0.1:45444)
0929 18:46:41.570381 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.570427 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 18:46:41.570640 30135 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:37257: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
I20260929 18:46:41.570796 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.570506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:37257 (local address 127.0.0.1:45448)
0929 18:46:41.570665 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.570723 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 18:46:41.571488 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.571308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:37257 (local address 127.0.0.1:45462)
0929 18:46:41.571393 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.571434 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 18:46:41.571599 30210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:37257: Network error: Client connection negotiation failed: client connection to 127.7.157.124:37257: connect: Connection refused (error 111)
W20260929 18:46:41.575946 30223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:41.575946 30222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:41.575953 30225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:41.576140 30216 server_base.cc:1034] running on GCE node
I20260929 18:46:41.576624 30216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:41.577623 30216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:41.578785 30216 hybrid_clock.cc:648] HybridClock initialized: now 1790707601578764 us; error 27 us; skew 500 ppm
Sep 29 18:46:41 dist-test-slave-smpn krb5kdc[30054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707601, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:41.581279 30216 init.cc:377] Logged in from keytab as kudu/127.7.157.124@KRBTEST.COM (short username kudu)
I20260929 18:46:41.582087 30216 webserver.cc:460] Webserver started at http://127.7.157.124:41891/ using document root <none> and password file <none>
I20260929 18:46:41.582592 30216 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:41.582654 30216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:41.582857 30216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:41.584851 30216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/data/instance:
uuid: "bdf3c80d0fdb4edeb494e5b77839d289"
format_stamp: "Formatted at 2026-09-29 18:46:41 on dist-test-slave-smpn"
server_key: "8754a3e6d3f4838a6cbd5a039a4a6d8e"
server_key_iv: "039a09d88a3fccb79b2e75181935b5b6"
server_key_version: "encryptionkey@0"
I20260929 18:46:41.585367 30216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/wal/instance:
uuid: "bdf3c80d0fdb4edeb494e5b77839d289"
format_stamp: "Formatted at 2026-09-29 18:46:41 on dist-test-slave-smpn"
server_key: "8754a3e6d3f4838a6cbd5a039a4a6d8e"
server_key_iv: "039a09d88a3fccb79b2e75181935b5b6"
server_key_version: "encryptionkey@0"
I20260929 18:46:41.588883 30216 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:46:41.591125 30232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:41.592298 30216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:41.592409 30216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/wal
uuid: "bdf3c80d0fdb4edeb494e5b77839d289"
format_stamp: "Formatted at 2026-09-29 18:46:41 on dist-test-slave-smpn"
server_key: "8754a3e6d3f4838a6cbd5a039a4a6d8e"
server_key_iv: "039a09d88a3fccb79b2e75181935b5b6"
server_key_version: "encryptionkey@0"
I20260929 18:46:41.592496 30216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:41.605486 30216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:41.608121 30216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:41.608287 30216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:41.614738 30216 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.124:37257
I20260929 18:46:41.614974 30284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.124:37257 every 8 connection(s)
I20260929 18:46:41.615677 30216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/data/info.pb
I20260929 18:46:41.618332 30285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:41.622049 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 30216
I20260929 18:46:41.622167 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707328618849-7797-0/minicluster-data/master-2/wal/instance
I20260929 18:46:41.621960 30285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:46:41.622440 7797 external_mini_cluster.cc:1428] Setting key ad7e89ccf9dea9a046977029b06047a4
Sep 29 18:46:41 dist-test-slave-smpn krb5kdc[30054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707601, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.124@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
Sep 29 18:46:41 dist-test-slave-smpn krb5kdc[30054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.124@KRBTEST.COM
I20260929 18:46:41.634372 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.623906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.7.157.126:45793)
0929 18:46:41.624013 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.624016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.624484 (+ 468us) server_negotiation.cc:407] Connection header received
0929 18:46:41.625245 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.625247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.625294 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.625361 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.626291 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.626784 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.627820 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.628077 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.628151 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.630687 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:41.630692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:41.630694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:41.630726 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:41.632194 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.633095 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.633098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.633099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.633172 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.633468 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.633471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.633473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.633661 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:41.633733 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:41.634096 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 18:46:41.634215 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:46:41.635202 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.623826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:45793 (local address 127.0.0.1:55340)
0929 18:46:41.624342 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.624386 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:41.625044 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:41.625424 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:41.625429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:41.625880 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:41.626153 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.626167 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.626879 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.626883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.627711 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.627720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.628202 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.628206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.628447 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.629022 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:41.629043 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:41.630545 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:41.632299 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.632307 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.632317 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.632729 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.633250 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.633253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.633255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.633378 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.633754 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:41.633760 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:41.633880 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 18:46:41.634042 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 18:46:41.634353 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":43,"spinlock_wait_cycles":53632,"thread_start_us":145,"threads_started":1}
I20260929 18:46:41.635373 30288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.623817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45476 (local address 127.7.157.124:37257)
0929 18:46:41.624239 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.624256 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.624294 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:46:41.625035 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.625059 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.625290 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.625450 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.626123 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.627100 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.627914 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.628398 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.628489 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.631146 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:41.631151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:41.631161 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:41.631189 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:41.632736 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.633078 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.633082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.633083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.633189 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.633459 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.633462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.633463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.633828 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:41.634009 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:41.634184 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 18:46:41.635195 (+ 1011us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":897,"server-negotiator.queue_time_us":290,"spinlock_wait_cycles":2432,"thread_start_us":133,"threads_started":1}
I20260929 18:46:41.636343 30285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:46:41 dist-test-slave-smpn krb5kdc[30054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707601, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.124@KRBTEST.COM for kudu/127.7.157.125@KRBTEST.COM
Sep 29 18:46:41 dist-test-slave-smpn krb5kdc[30054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:41.644069 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.636791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:43667 (local address 127.0.0.1:55210)
0929 18:46:41.636937 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.636950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:41.637017 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:41.637235 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:41.637237 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:41.637379 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:41.637473 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.637479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.638330 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.638332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.639183 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.639193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.639606 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.639609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.639708 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.640092 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:41.640106 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:41.641234 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:41.642797 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.642799 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.642801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.642963 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.643248 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.643251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.643253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.643320 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.643647 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:41.643650 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:41.643726 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 18:46:41.643806 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:46:41.643921 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 18:46:41.644098 30215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.636884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.7.157.125:43667)
0929 18:46:41.637005 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.637009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.637023 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:46:41.637099 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.637102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.637145 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.637213 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.637605 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.638213 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.639286 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.639509 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.639594 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.641399 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:41.641403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:41.641405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:41.641427 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:41.642711 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.643054 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.643057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.643059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.643166 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.643397 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.643400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.643401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.643566 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:41.643646 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:41.643867 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 18:46:41.643968 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:46:41.644933 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.637570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55354 (local address 127.7.157.126:45793)
0929 18:46:41.637678 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.637682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.637694 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:46:41.637730 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.637733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.637771 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.637842 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.638332 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.639059 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.639785 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.640041 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.640130 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.642155 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:41.642158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:41.642159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:41.642187 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:41.643448 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.643804 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.643807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.643809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.643882 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.644124 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.644127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.644128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.644340 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:41.644550 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:41.644692 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 18:46:41.644796 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:46:41.645193 30285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:46:41 dist-test-slave-smpn krb5kdc[30054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707601, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.124@KRBTEST.COM for kudu/127.7.157.124@KRBTEST.COM
Sep 29 18:46:41 dist-test-slave-smpn krb5kdc[30054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.125@KRBTEST.COM
I20260929 18:46:41.652493 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.645636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:37257 (local address 127.0.0.1:45484)
0929 18:46:41.645716 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.645726 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:41.645795 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:41.646068 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:41.646070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:41.646218 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:41.646293 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.646299 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.647018 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.647021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.647619 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.647628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.648012 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.648015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.648090 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.648450 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:41.648464 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:41.649577 (+ 1113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:41.651117 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.651120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.651122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.651341 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.651580 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.651582 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.651584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.651668 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.652013 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:41.652016 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:41.652091 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 18:46:41.652183 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 18:46:41.652310 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 18:46:41.652499 30288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.645750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.7.157.124:37257)
0929 18:46:41.645835 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.645838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.645850 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:46:41.645879 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.645882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.645919 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.645984 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.646397 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.646922 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.647718 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.647928 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.647996 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.649694 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:41.649698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:41.649700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:41.649721 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:41.650999 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.651426 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.651429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.651430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.651496 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.651775 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.651776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.651777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.651924 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:41.652004 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:41.652221 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 18:46:41.652363 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":23}
I20260929 18:46:41.653295 30215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.645971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.7.157.125:43667)
0929 18:46:41.646070 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.646074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.646085 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:46:41.646180 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.646181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.646212 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.646267 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.646801 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.647375 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.648206 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.648400 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.648464 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.650471 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:41.650475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:41.650487 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:41.650510 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:41.651905 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.652250 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.652254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.652255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.652326 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.652551 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.652554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.652555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.652735 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:41.652971 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:41.653080 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 18:46:41.653169 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:46:41.656340 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:41.656461 30285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289: Bootstrap starting.
I20260929 18:46:41.657655 30109 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55354
I20260929 18:46:41.658936 30285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:41.659667 30285 log.cc:824] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:41.661602 30285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289: No bootstrap required, opened a new log
I20260929 18:46:41.662729 30210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } attempt: 3
I20260929 18:46:41.665223 30285 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } }
I20260929 18:46:41.665431 30285 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf3c80d0fdb4edeb494e5b77839d289, State: Initialized, Role: FOLLOWER
I20260929 18:46:41.665735 30285 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } }
I20260929 18:46:41.666232 30291 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } } }
I20260929 18:46:41.666447 30291 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:46:41.667013 30285 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 18:46:41 dist-test-slave-smpn krb5kdc[30054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707601, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.125@KRBTEST.COM for kudu/127.7.157.124@KRBTEST.COM
I20260929 18:46:41.669929 30135 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } attempt: 3
I20260929 18:46:41.671881 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.663186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:37257 (local address 127.0.0.1:45498)
0929 18:46:41.663324 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.663338 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:41.663423 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:41.663674 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:41.663677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:41.663828 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:41.663915 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.663921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.664865 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.664869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.665696 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.665705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.666237 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.666241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.666326 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.666831 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:41.666846 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:41.668313 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:41.670234 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.670237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.670239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.670480 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.670789 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.670792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.670794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.670896 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.671386 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:41.671389 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:41.671500 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 18:46:41.671594 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 18:46:41.671723 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 18:46:41.672402 30288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.663254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.7.157.124:37257)
0929 18:46:41.663375 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.663380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.663398 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:46:41.663510 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.663513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.663558 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.663640 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.664022 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.664762 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.665809 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.666116 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.666212 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.668446 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:41.668450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:41.668452 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:41.668476 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:41.670108 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.670607 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.670611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.670613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.670695 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.670997 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.671000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.671002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.671259 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:41.671343 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:41.672163 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 18:46:41.672272 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 18:46:41.672856 30302 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:46:41.672915 30302 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:46:41.670768 30216 master_runner.cc:186] Error getting master registration for 127.7.157.126:45793: OK, instance_id { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" instance_seqno: 1790707601207622 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:46:41.674463 30216 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 29 18:46:41 dist-test-slave-smpn krb5kdc[30054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707601, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for kudu/127.7.157.124@KRBTEST.COM
I20260929 18:46:41.677373 30210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c: Bootstrap starting.
I20260929 18:46:41.679669 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.670281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:37257 (local address 127.0.0.1:45514)
0929 18:46:41.670394 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:41.670408 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:41.670482 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:41.670947 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:41.670949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:41.671149 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:41.671270 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.671277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.672560 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.672563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.673355 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:41.673366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.673968 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.673972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:41.674057 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.674534 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:41.674548 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:41.676017 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:41.678080 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.678084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.678087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.678329 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.678632 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:41.678635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:41.678637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:41.678726 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:41.679137 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:41.679140 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:41.679254 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 18:46:41.679356 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 18:46:41.679502 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 18:46:41.679735 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.670459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.7.157.124:37257)
0929 18:46:41.670712 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.670722 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.670733 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:46:41.670762 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.670765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.670821 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.671372 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.671487 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.672427 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.673494 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.673840 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.673934 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.676161 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:41.676165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:41.676168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:41.676221 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:41.677984 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.678429 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.678432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.678434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.678539 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.678815 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.678818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.678820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.679016 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:41.679146 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:41.679457 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 18:46:41.679568 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
I20260929 18:46:41.680227 30210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:41.681061 30210 log.cc:824] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:41.683143 30135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec: Bootstrap starting.
I20260929 18:46:41.683398 30210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c: No bootstrap required, opened a new log
I20260929 18:46:41.685797 30135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:41.686656 30135 log.cc:824] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:41.687280 30210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } }
I20260929 18:46:41.687469 30210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41038e49760c490d96dbea59e4f13d4c, State: Initialized, Role: FOLLOWER
I20260929 18:46:41.687739 30210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } }
I20260929 18:46:41.688284 30307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } } }
I20260929 18:46:41.688611 30307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:46:41.689069 30210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:41.689088 30135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec: No bootstrap required, opened a new log
I20260929 18:46:41.692193 30135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } }
I20260929 18:46:41.692425 30135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ccfe34fb2b44dadba92d2d33a0dd9ec, State: Initialized, Role: FOLLOWER
I20260929 18:46:41.692756 30135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } }
I20260929 18:46:41.693280 30319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } } }
I20260929 18:46:41.693540 30319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:46:41.694080 30135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:41.694167 30109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 18:46:41.694264 30109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:41.695473 30109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } }
I20260929 18:46:41.696378 30109 leader_election.cc:288] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [CANDIDATE]: Term 1 election: Requested vote from peers 41038e49760c490d96dbea59e4f13d4c (127.7.157.125:43667), bdf3c80d0fdb4edeb494e5b77839d289 (127.7.157.124:37257)
I20260929 18:46:41.692683 30142 master_runner.cc:186] Error getting master registration for 127.7.157.126:45793: OK, instance_id { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" instance_seqno: 1790707601207622 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:46:41.696537 30142 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:46:41.697325 30184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "41038e49760c490d96dbea59e4f13d4c"
I20260929 18:46:41.697290 30259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bdf3c80d0fdb4edeb494e5b77839d289"
I20260929 18:46:41.697525 30184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:41.697557 30259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:41.698758 30184 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ccfe34fb2b44dadba92d2d33a0dd9ec in term 1.
I20260929 18:46:41.698771 30259 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ccfe34fb2b44dadba92d2d33a0dd9ec in term 1.
W20260929 18:46:41.698894 30318 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:46:41.698942 30318 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:46:41.699201 30083 leader_election.cc:302] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41038e49760c490d96dbea59e4f13d4c, 7ccfe34fb2b44dadba92d2d33a0dd9ec; no voters:
I20260929 18:46:41.699481 30319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:41.700187 30319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [term 1 LEADER]: Becoming Leader. State: Replica: 7ccfe34fb2b44dadba92d2d33a0dd9ec, State: Running, Role: LEADER
I20260929 18:46:41.700891 30319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } }
I20260929 18:46:41.703241 30336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ccfe34fb2b44dadba92d2d33a0dd9ec. Latest consensus state: current_term: 1 leader_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } } }
I20260929 18:46:41.703372 30336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [sys.catalog]: This master's current role is: LEADER
W20260929 18:46:41.704710 30088 tablet.cc:2369] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:46:41.708173 30335 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:46:41.708251 30335 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:46:41.708319 30338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:46:41.709393 30338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:41.720480 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.705561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.7.157.124:37257)
0929 18:46:41.705691 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.705696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.711400 (+ 5704us) server_negotiation.cc:407] Connection header received
0929 18:46:41.711594 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.711597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.711645 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.711692 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.712640 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.713442 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.714627 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.715015 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.715125 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.716936 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:41.716940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:41.716943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:41.716970 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:41.718601 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.719116 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.719120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.719122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.719208 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.719492 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.719495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.719497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.719720 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:41.719831 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:41.720168 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 18:46:41.720314 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:46:41.727751 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.706248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.7.157.126:45793)
0929 18:46:41.706345 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.706349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.712950 (+ 6601us) server_negotiation.cc:407] Connection header received
0929 18:46:41.719291 (+ 6341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.719294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.719344 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.719392 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.720280 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.721092 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.722109 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.722411 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.722482 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.724183 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:41.724187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:41.724190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:41.724221 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:41.725941 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.726395 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.726398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.726400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.726482 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.726769 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.726772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.726774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.726986 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:41.727116 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:41.727446 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 18:46:41.727585 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 18:46:41.729966 30215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:41.711544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.7.157.125:43667)
0929 18:46:41.711654 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.711659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.721270 (+ 9611us) server_negotiation.cc:407] Connection header received
0929 18:46:41.721432 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.721435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.721483 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.721530 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.722335 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.723113 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.724188 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:41.724521 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:41.724600 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:41.726299 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:41.726303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:41.726306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:41.726334 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:41.727926 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.728634 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.728637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.728639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.728741 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:41.729027 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:41.729030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:41.729032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:41.729241 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:41.729337 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:41.729688 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 18:46:41.729806 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:46:41.741047 30184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c [term 1 FOLLOWER]: Refusing update from remote peer 7ccfe34fb2b44dadba92d2d33a0dd9ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:46:41.742484 30259 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289 [term 1 FOLLOWER]: Refusing update from remote peer 7ccfe34fb2b44dadba92d2d33a0dd9ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:46:41.743364 30319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:46:41.743808 30319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:46:41.752871 30307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ccfe34fb2b44dadba92d2d33a0dd9ec. Latest consensus state: current_term: 1 leader_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } } }
I20260929 18:46:41.753024 30307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:46:41.754823 30291 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ccfe34fb2b44dadba92d2d33a0dd9ec. Latest consensus state: current_term: 1 leader_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } } }
I20260929 18:46:41.754940 30291 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:46:41.759269 30336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } } }
I20260929 18:46:41.759415 30336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:41.760627 30291 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } } }
I20260929 18:46:41.760759 30291 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:46:41.763315 30343 mvcc.cc:205] Tried to move back new op lower bound from 7334738336686071808 to 7334738336570351616. Current Snapshot: MvccSnapshot[applied={T|T < 7334738336686071808}]
I20260929 18:46:41.765923 30319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } } }
I20260929 18:46:41.766057 30319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:41.767048 30307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccfe34fb2b44dadba92d2d33a0dd9ec" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 45793 } } peers { permanent_uuid: "41038e49760c490d96dbea59e4f13d4c" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 43667 } } peers { permanent_uuid: "bdf3c80d0fdb4edeb494e5b77839d289" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 37257 } } }
I20260929 18:46:41.767177 30307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:46:41.767905 30345 mvcc.cc:205] Tried to move back new op lower bound from 7334738336686071808 to 7334738336570351616. Current Snapshot: MvccSnapshot[applied={T|T < 7334738336686071808}]
I20260929 18:46:41.775188 30338 catalog_manager.cc:1347] Generated new cluster ID: 70e708d0bf5147b494cabc930d62ec93
I20260929 18:46:41.775262 30338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:41.798219 30338 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:41.799549 30338 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:41.811936 30338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ccfe34fb2b44dadba92d2d33a0dd9ec: Generated new TSK 0
I20260929 18:46:41.812793 30338 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:46:41.844153 30138 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:41.841745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.7.157.126:45793)
0929 18:46:41.841854 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.841858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.843161 (+ 1303us) server_negotiation.cc:407] Connection header received
0929 18:46:41.843343 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.843346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.843395 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.843443 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.844020 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55370: BlockingRecv error: recv got EOF from 127.0.0.1:55370 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 18:46:41.845654 30215 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:41.841318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.7.157.125:43667)
0929 18:46:41.841440 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.841444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.844735 (+ 3291us) server_negotiation.cc:407] Connection header received
0929 18:46:41.844952 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.844955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.845015 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.845064 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.845531 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55232: BlockingRecv error: recv got EOF from 127.0.0.1:55232 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 18:46:41.846933 30303 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:41.842142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.7.157.124:37257)
0929 18:46:41.842275 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.842280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.846107 (+ 3827us) server_negotiation.cc:407] Connection header received
0929 18:46:41.846255 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.846258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.846304 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.846350 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.846820 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45532: BlockingRecv error: recv got EOF from 127.0.0.1:45532 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 18:46:41.851356 30215 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:41.849817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.7.157.125:43667)
0929 18:46:41.849917 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.849921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.850438 (+ 517us) server_negotiation.cc:407] Connection header received
0929 18:46:41.850630 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.850633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.850706 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.850754 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.851263 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55242: BlockingRecv error: recv got EOF from 127.0.0.1:55242 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 18:46:41.856201 30215 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:41.854728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.7.157.125:43667)
0929 18:46:41.854826 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.854829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.855349 (+ 520us) server_negotiation.cc:407] Connection header received
0929 18:46:41.855523 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.855526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.855575 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.855623 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.856105 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55248: BlockingRecv error: recv got EOF from 127.0.0.1:55248 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 18:46:41.866709 30215 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:41.864956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.7.157.125:43667)
0929 18:46:41.865083 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.865088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.865704 (+ 616us) server_negotiation.cc:407] Connection header received
0929 18:46:41.865899 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.865903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.865955 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.866018 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.866581 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55252: BlockingRecv error: recv got EOF from 127.0.0.1:55252 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 18:46:41.874890 30215 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:41.873247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.7.157.125:43667)
0929 18:46:41.873389 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.873394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.873885 (+ 491us) server_negotiation.cc:407] Connection header received
0929 18:46:41.874107 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.874111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.874165 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.874219 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.874770 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55266: BlockingRecv error: recv got EOF from 127.0.0.1:55266 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 18:46:41.887184 30215 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:41.885433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.7.157.125:43667)
0929 18:46:41.885549 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.885553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.886092 (+ 539us) server_negotiation.cc:407] Connection header received
0929 18:46:41.886306 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.886310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.886368 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.886423 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.887018 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55272: BlockingRecv error: recv got EOF from 127.0.0.1:55272 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 18:46:41.906746 30215 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:41.905549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.7.157.125:43667)
0929 18:46:41.905678 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.905683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.905931 (+ 248us) server_negotiation.cc:407] Connection header received
0929 18:46:41.906077 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.906079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.906127 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.906205 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.906626 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55274: BlockingRecv error: recv got EOF from 127.0.0.1:55274 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 18:46:41.941943 30215 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:41.940793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55280 (local address 127.7.157.125:43667)
0929 18:46:41.940959 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:41.940964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:41.941085 (+ 121us) server_negotiation.cc:407] Connection header received
0929 18:46:41.941230 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:41.941234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:41.941284 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:41.941353 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:41.941819 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55280: BlockingRecv error: recv got EOF from 127.0.0.1:55280 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 18:46:42.009027 30215 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:42.007864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55292 (local address 127.7.157.125:43667)
0929 18:46:42.008024 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:42.008029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:42.008170 (+ 141us) server_negotiation.cc:407] Connection header received
0929 18:46:42.008317 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:42.008320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:42.008370 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:42.008439 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:42.008896 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55292: BlockingRecv error: recv got EOF from 127.0.0.1:55292 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 18:46:42.140209 30215 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:42.139028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.7.157.125:43667)
0929 18:46:42.139202 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:42.139207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:42.139321 (+ 114us) server_negotiation.cc:407] Connection header received
0929 18:46:42.139464 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:42.139503 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:42.139552 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:42.139629 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:42.140073 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55306: BlockingRecv error: recv got EOF from 127.0.0.1:55306 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 18:46:42.399114 30215 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:42.398144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.7.157.125:43667)
0929 18:46:42.398275 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:42.398279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:42.398365 (+ 86us) server_negotiation.cc:407] Connection header received
0929 18:46:42.398488 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:42.398490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:42.398537 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:42.398616 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:42.398975 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55320: BlockingRecv error: recv got EOF from 127.0.0.1:55320 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:46:42.674189 30302 catalog_manager.cc:1259] Loaded cluster ID: 70e708d0bf5147b494cabc930d62ec93
I20260929 18:46:42.674274 30302 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289: loading cluster ID for follower catalog manager: success
I20260929 18:46:42.676038 30302 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289: acquiring CA information for follower catalog manager: success
I20260929 18:46:42.677228 30302 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bdf3c80d0fdb4edeb494e5b77839d289: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:46:42.700177 30318 catalog_manager.cc:1259] Loaded cluster ID: 70e708d0bf5147b494cabc930d62ec93
I20260929 18:46:42.700277 30318 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c: loading cluster ID for follower catalog manager: success
I20260929 18:46:42.702047 30318 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c: acquiring CA information for follower catalog manager: success
I20260929 18:46:42.703414 30318 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 41038e49760c490d96dbea59e4f13d4c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:46:42.917333 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:42.913442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.7.157.125:43667)
0929 18:46:42.913673 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:42.913678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:42.913772 (+ 94us) server_negotiation.cc:407] Connection header received
0929 18:46:42.913924 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:42.913928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:42.914052 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:42.914158 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:46:42.914428 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:42.914956 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:42.915842 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:42.916068 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:42.916164 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:42.916485 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:46:42.916929 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:46:42.917045 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:42.917119 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 18:46:42.917180 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":65,"threads_started":1}
I20260929 18:46:42.923493 30421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:42.920137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45546 (local address 127.7.157.124:37257)
0929 18:46:42.920345 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:42.920349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:42.920362 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:46:42.920482 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:42.920484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:42.920598 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:42.920699 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:46:42.920853 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:42.921352 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:42.922138 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:42.922343 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:42.922406 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:42.922727 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:46:42.923168 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:46:42.923259 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:42.923331 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 18:46:42.923377 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":68,"threads_started":1}
I20260929 18:46:42.930128 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:42.926334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.7.157.125:43667)
0929 18:46:42.926454 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:42.926458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:42.926973 (+ 515us) server_negotiation.cc:407] Connection header received
0929 18:46:42.927088 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:42.927091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:42.927141 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:42.927202 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:46:42.927511 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:42.928131 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:42.929012 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:42.929338 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:42.929422 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:42.929706 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:46:42.929782 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:46:42.929850 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:42.929942 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 18:46:42.929999 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:46:42.930371 30421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:42.926351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.7.157.124:37257)
0929 18:46:42.926441 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:42.926446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:42.926728 (+ 282us) server_negotiation.cc:407] Connection header received
0929 18:46:42.926815 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:42.926818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:42.926863 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:42.926922 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:46:42.927225 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:42.927822 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:42.928987 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:42.929309 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:42.929398 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:42.929731 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:46:42.929789 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:46:42.929843 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:42.930106 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 18:46:42.930152 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:46:42.930496 30429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:42.926088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55378 (local address 127.7.157.126:45793)
0929 18:46:42.926312 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:42.926317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:42.926498 (+ 181us) server_negotiation.cc:407] Connection header received
0929 18:46:42.926613 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:42.926615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:42.926724 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:42.926828 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:46:42.927090 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:42.927755 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:42.928974 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:42.929210 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:42.929295 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:42.929595 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:46:42.929898 (+ 303us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:46:42.929972 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:42.930307 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 18:46:42.930361 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":63,"threads_started":1}
I20260929 18:46:42.933403 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 30067
I20260929 18:46:42.939121 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 30142
I20260929 18:46:42.943830 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 30216
2026-09-29T18:46:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2984 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:46:42 dist-test-slave-smpn krb5kdc[30432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:46:42 dist-test-slave-smpn krb5kdc[30432](info): set up 2 sockets
Sep 29 18:46:42 dist-test-slave-smpn krb5kdc[30432](info): commencing operation
krb5kdc: starting...
W20260929 18:46:43.911101 7797 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/test-user.keytab.
Sep 29 18:46:43 dist-test-slave-smpn krb5kdc[30432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:46:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:46:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.157.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
Entry for principal kudu/127.7.157.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab.
I20260929 18:46:44.023880 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:46421
--webserver_interface=127.7.157.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:33019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:46421,127.7.157.125:35813,127.7.157.124:40827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:46:44.134533 30445 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:46:44.134843 30445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:44.134867 30445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:44.134915 30445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:44.138967 30445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:44.139035 30445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:44.139053 30445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:44.139093 30445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:44.139115 30445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:44.144589 30445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:33019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:46421,127.7.157.125:35813,127.7.157.124:40827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.126.keytab
--principal=kudu/127.7.157.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.126:46421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.157.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:44.145764 30445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:44.146857 30445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:44.155545 30453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:44.155545 30450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:44.155622 30451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:44.155845 30445 server_base.cc:1034] running on GCE node
I20260929 18:46:44.156445 30445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:44.157588 30445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:44.158744 30445 hybrid_clock.cc:648] HybridClock initialized: now 1790707604158729 us; error 26 us; skew 500 ppm
Sep 29 18:46:44 dist-test-slave-smpn krb5kdc[30432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707604, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:44.161441 30445 init.cc:377] Logged in from keytab as kudu/127.7.157.126@KRBTEST.COM (short username kudu)
I20260929 18:46:44.162353 30445 webserver.cc:460] Webserver started at http://127.7.157.126:36621/ using document root <none> and password file <none>
I20260929 18:46:44.162905 30445 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:44.162967 30445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:44.163214 30445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:44.165300 30445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/data/instance:
uuid: "dae107ceee36468aae644236df069a7e"
format_stamp: "Formatted at 2026-09-29 18:46:44 on dist-test-slave-smpn"
server_key: "f2f6ce6f9e98fa234490b1c167d1b1bd"
server_key_iv: "5d45384e6c2f6110e35e3d5731706c69"
server_key_version: "encryptionkey@0"
I20260929 18:46:44.165910 30445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/wal/instance:
uuid: "dae107ceee36468aae644236df069a7e"
format_stamp: "Formatted at 2026-09-29 18:46:44 on dist-test-slave-smpn"
server_key: "f2f6ce6f9e98fa234490b1c167d1b1bd"
server_key_iv: "5d45384e6c2f6110e35e3d5731706c69"
server_key_version: "encryptionkey@0"
I20260929 18:46:44.169725 30445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:46:44.172200 30460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:44.173499 30445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:46:44.173621 30445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/wal
uuid: "dae107ceee36468aae644236df069a7e"
format_stamp: "Formatted at 2026-09-29 18:46:44 on dist-test-slave-smpn"
server_key: "f2f6ce6f9e98fa234490b1c167d1b1bd"
server_key_iv: "5d45384e6c2f6110e35e3d5731706c69"
server_key_version: "encryptionkey@0"
I20260929 18:46:44.173725 30445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:44.189071 30445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:44.191776 30445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:44.191951 30445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:44.198216 30445 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.126:46421
I20260929 18:46:44.198464 30512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.126:46421 every 8 connection(s)
I20260929 18:46:44.199144 30445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/data/info.pb
I20260929 18:46:44.199956 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 30445
I20260929 18:46:44.200062 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-0/wal/instance
I20260929 18:46:44.200336 7797 external_mini_cluster.cc:1428] Setting key d8dce445b4b2d0096eba9beb4dfb9b97
I20260929 18:46:44.202028 30513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:44.207050 30513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.157.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.125@KRBTEST.COM" created.
Sep 29 18:46:44 dist-test-slave-smpn krb5kdc[30432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707604, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:44.222795 30516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.209774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42524 (local address 127.7.157.126:46421)
0929 18:46:44.210242 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.210261 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.210487 (+ 226us) server_negotiation.cc:407] Connection header received
0929 18:46:44.211700 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.211705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.211902 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.212046 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.212959 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.214111 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.215112 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.215705 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.215879 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.218395 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:44.218400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:44.218411 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:44.218436 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:44.219976 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.220514 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.220519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.220520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.220631 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.220895 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.220897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.220898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.221337 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:44.221451 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:44.221850 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 18:46:44.222045 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":306,"thread_start_us":136,"threads_started":1}
I20260929 18:46:44.222798 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.209774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:46421 (local address 127.0.0.1:42524)
0929 18:46:44.210336 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.210386 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:44.211315 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:44.212088 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:44.212094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:44.212557 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:44.212821 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.212834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.214256 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.214259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.214976 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.214985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.215889 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.215893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.216010 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.216604 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:44.216622 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:44.218241 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:44.220078 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.220086 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.220094 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.220387 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.220716 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.220718 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.220719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.220827 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.221431 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:44.221436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:44.221558 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 18:46:44.221701 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 18:46:44.221958 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"spinlock_wait_cycles":56704,"thread_start_us":152,"threads_started":1}
I20260929 18:46:44.225483 30513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.125.keytab.
Entry for principal kudu/127.7.157.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.125.keytab.
I20260929 18:46:44.226202 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.225954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:35813 (local address 127.0.0.1:36784)
0929 18:46:44.226074 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.226141 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 18:46:44.226382 30461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.157.125:35813: connect: Connection refused (error 111)
I20260929 18:46:44.226516 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.125:35813
--webserver_interface=127.7.157.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.125.keytab
--principal=kudu/127.7.157.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:33019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:46421,127.7.157.125:35813,127.7.157.124:40827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:46:44.227460 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.227249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:35813 (local address 127.0.0.1:36800)
0929 18:46:44.227355 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.227397 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 18:46:44.227579 30513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.125:35813: Network error: Client connection negotiation failed: client connection to 127.7.157.125:35813: connect: Connection refused (error 111)
I20260929 18:46:44.276847 30513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } attempt: 1
I20260929 18:46:44.277696 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.277398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:35813 (local address 127.0.0.1:36810)
0929 18:46:44.277539 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.277613 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 18:46:44.278438 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.278220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:35813 (local address 127.0.0.1:36814)
0929 18:46:44.278308 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.278353 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 18:46:44.278589 30513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.125:35813: Network error: Client connection negotiation failed: client connection to 127.7.157.125:35813: connect: Connection refused (error 111)
W20260929 18:46:44.337420 30520 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:46:44.337716 30520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:44.337761 30520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:44.337823 30520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:44.341866 30520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:44.341923 30520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:44.341948 30520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:44.341970 30520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:44.341998 30520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:44.346774 30513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } attempt: 2
I20260929 18:46:44.347334 30520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:33019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:46421,127.7.157.125:35813,127.7.157.124:40827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.125.keytab
--principal=kudu/127.7.157.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.125:35813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.157.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:44.347707 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.347348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:35813 (local address 127.0.0.1:36828)
0929 18:46:44.347550 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.347631 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 18:46:44.348379 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.348182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:35813 (local address 127.0.0.1:36844)
0929 18:46:44.348279 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.348325 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.125:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 18:46:44.348517 30513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.125:35813: Network error: Client connection negotiation failed: client connection to 127.7.157.125:35813: connect: Connection refused (error 111)
I20260929 18:46:44.348623 30520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:44.349685 30520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:46:44.357908 30526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:44.357946 30525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:44.357947 30528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:44.358209 30520 server_base.cc:1034] running on GCE node
I20260929 18:46:44.358762 30520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:44.359958 30520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:44.361156 30520 hybrid_clock.cc:648] HybridClock initialized: now 1790707604361124 us; error 34 us; skew 500 ppm
Sep 29 18:46:44 dist-test-slave-smpn krb5kdc[30432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707604, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:44.363996 30520 init.cc:377] Logged in from keytab as kudu/127.7.157.125@KRBTEST.COM (short username kudu)
I20260929 18:46:44.364874 30520 webserver.cc:460] Webserver started at http://127.7.157.125:38065/ using document root <none> and password file <none>
I20260929 18:46:44.365407 30520 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:44.365474 30520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:44.365651 30520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:44.367765 30520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/data/instance:
uuid: "5a99d6e003844e2bae47bd17687fa342"
format_stamp: "Formatted at 2026-09-29 18:46:44 on dist-test-slave-smpn"
server_key: "9b911dbec602673cd5f7d9183efdacf0"
server_key_iv: "4fd0e73098b5243bf2b9cae36b28757b"
server_key_version: "encryptionkey@0"
I20260929 18:46:44.368317 30520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/wal/instance:
uuid: "5a99d6e003844e2bae47bd17687fa342"
format_stamp: "Formatted at 2026-09-29 18:46:44 on dist-test-slave-smpn"
server_key: "9b911dbec602673cd5f7d9183efdacf0"
server_key_iv: "4fd0e73098b5243bf2b9cae36b28757b"
server_key_version: "encryptionkey@0"
I20260929 18:46:44.372073 30520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:46:44.374573 30535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:44.375864 30520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:44.375993 30520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/wal
uuid: "5a99d6e003844e2bae47bd17687fa342"
format_stamp: "Formatted at 2026-09-29 18:46:44 on dist-test-slave-smpn"
server_key: "9b911dbec602673cd5f7d9183efdacf0"
server_key_iv: "4fd0e73098b5243bf2b9cae36b28757b"
server_key_version: "encryptionkey@0"
I20260929 18:46:44.376093 30520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:44.394482 30520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:44.397549 30520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:44.397756 30520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:44.404326 30520 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.125:35813
I20260929 18:46:44.404564 30587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.125:35813 every 8 connection(s)
I20260929 18:46:44.405295 30520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/data/info.pb
I20260929 18:46:44.408061 30588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:44.411754 30588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:46:44.412448 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 30520
I20260929 18:46:44.412571 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-1/wal/instance
I20260929 18:46:44.412874 7797 external_mini_cluster.cc:1428] Setting key b1bb3794ec284d16ffddf33214d786da
WARNING: no policy specified for kudu/127.7.157.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.157.124@KRBTEST.COM" created.
Sep 29 18:46:44 dist-test-slave-smpn krb5kdc[30432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707604, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.125@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:44.427479 30516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.413984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.7.157.126:46421)
0929 18:46:44.414135 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.414147 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.415039 (+ 892us) server_negotiation.cc:407] Connection header received
0929 18:46:44.416109 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.416112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.416174 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.416262 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.417492 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.418317 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.419505 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.419893 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.420005 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.424096 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:44.424099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:44.424102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:44.424128 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:44.425452 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.425976 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.425979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.425980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.426055 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.426375 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.426378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.426379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.426550 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:44.426633 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:44.427192 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 18:46:44.427314 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:46:44.428279 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.414337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:46421 (local address 127.0.0.1:42538)
0929 18:46:44.414848 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.414906 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:44.415886 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:44.416408 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:44.416415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:44.416844 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:44.417329 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.417344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.418446 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.418449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.419382 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.419393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.420042 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.420044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.420199 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.420730 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:44.420745 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:44.423909 (+ 3164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:44.425568 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.425574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.425592 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.425872 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.426135 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.426138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.426140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.426281 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.426648 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:44.426655 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:44.426918 (+ 263us) client_negotiation.cc:769] Sending connection context
0929 18:46:44.427100 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 18:46:44.427434 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":123,"threads_started":1}
I20260929 18:46:44.429711 30588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:46:44 dist-test-slave-smpn krb5kdc[30432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707604, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.125@KRBTEST.COM for kudu/127.7.157.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.157.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.124.keytab.
Entry for principal kudu/127.7.157.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.124.keytab.
I20260929 18:46:44.437810 7797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
/tmp/dist-test-taskLJQtbs/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.124:40827
--webserver_interface=127.7.157.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.124.keytab
--principal=kudu/127.7.157.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.157.84:33019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.157.126:46421,127.7.157.125:35813,127.7.157.124:40827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:46:44.439673 30513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } attempt: 3
I20260929 18:46:44.440083 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.430225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:35813 (local address 127.0.0.1:36848)
0929 18:46:44.430378 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.430388 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:44.430445 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:44.430974 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:44.430977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:44.431157 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:44.431246 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.431252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.432266 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.432269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.432809 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.432818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.433396 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.433399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.433483 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.433992 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:44.434007 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:44.435316 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:44.436986 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.436989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.436991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.437234 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.439089 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.439092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.439094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.439186 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.439611 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:44.439614 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:44.439719 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 18:46:44.439797 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:46:44.439925 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 18:46:44.440613 30593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.430353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.7.157.125:35813)
0929 18:46:44.430651 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.430661 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.430679 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:46:44.430708 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.430713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.430854 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.430968 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.431342 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.432168 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.432908 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.433299 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.433389 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.435463 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:44.435472 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:44.435493 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:44.435524 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:44.436896 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.438890 (+ 1994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.438895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.438896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.438973 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.439280 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.439283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.439285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.439514 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:44.439630 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:44.439946 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 18:46:44.440069 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":124,"threads_started":1}
I20260929 18:46:44.442559 30588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:46:44.443251 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.442997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:40827 (local address 127.0.0.1:60782)
0929 18:46:44.443121 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.443198 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 18:46:44.443832 30539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
Sep 29 18:46:44 dist-test-slave-smpn krb5kdc[30432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707604, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for kudu/127.7.157.125@KRBTEST.COM
I20260929 18:46:44.445299 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.444672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:40827 (local address 127.0.0.1:60792)
0929 18:46:44.445185 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.445236 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433}
W20260929 18:46:44.445459 30588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:40827: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
I20260929 18:46:44.448637 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.440078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:35813 (local address 127.0.0.1:36856)
0929 18:46:44.440224 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.440239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:44.440316 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:44.440844 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:44.440847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:44.441010 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:44.441112 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.441119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.441986 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.441989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.442596 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.442607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.443415 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.443419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.443539 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.444013 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:44.444027 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:44.445371 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:44.447344 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.447347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.447349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.447534 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.447802 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.447804 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.447805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.447865 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.448252 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:44.448254 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:44.448335 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 18:46:44.448414 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:46:44.448521 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 18:46:44.448768 30595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.440342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.7.157.125:35813)
0929 18:46:44.440609 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.440617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.440629 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:46:44.440659 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.440662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.440705 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.441226 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.441320 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.441882 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.442698 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.442989 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.443766 (+ 777us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.445701 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:44.445705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:44.445708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:44.445733 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:44.447242 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.447625 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.447628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.447630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.447719 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.447937 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.447940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.447942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.448171 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:44.448273 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:44.448489 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 18:46:44.448604 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":97,"threads_started":1}
I20260929 18:46:44.449431 30513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:46:44.450099 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.449876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:40827 (local address 127.0.0.1:60798)
0929 18:46:44.449988 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.450038 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 18:46:44.450702 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.450487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:40827 (local address 127.0.0.1:60812)
0929 18:46:44.450599 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.450638 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 18:46:44.450832 30513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:40827: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
I20260929 18:46:44.481957 30513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } attempt: 1
I20260929 18:46:44.482816 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.482517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:40827 (local address 127.0.0.1:60828)
0929 18:46:44.482668 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.482735 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 18:46:44.483529 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.483320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:40827 (local address 127.0.0.1:60834)
0929 18:46:44.483420 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.483469 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 18:46:44.483649 30513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:40827: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
I20260929 18:46:44.494750 30588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } attempt: 1
I20260929 18:46:44.495674 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.495344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:40827 (local address 127.0.0.1:60836)
0929 18:46:44.495520 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.495592 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 18:46:44.496419 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.496188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:40827 (local address 127.0.0.1:60848)
0929 18:46:44.496311 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.496362 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 18:46:44.496574 30588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:40827: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
W20260929 18:46:44.549817 30594 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:46:44.550154 30594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:46:44.550194 30594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:46:44.550257 30594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:46:44.554332 30594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:46:44.554399 30594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:46:44.554426 30594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:46:44.554446 30594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:46:44.554468 30594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:46:44.558815 30513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } attempt: 2
I20260929 18:46:44.559727 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.559401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:40827 (local address 127.0.0.1:60864)
0929 18:46:44.559580 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.559647 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 18:46:44.560149 30594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.157.84:33019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.157.126:46421,127.7.157.125:35813,127.7.157.124:40827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/krb5kdc/kudu_127.7.157.124.keytab
--principal=kudu/127.7.157.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.157.124:40827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.157.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:46:44.560442 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.560216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:40827 (local address 127.0.0.1:60874)
0929 18:46:44.560319 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.560376 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 18:46:44.560556 30513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:40827: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
I20260929 18:46:44.561313 30594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:46:44.562321 30594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:46:44.564754 30588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } attempt: 2
I20260929 18:46:44.565537 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.565238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:40827 (local address 127.0.0.1:60888)
0929 18:46:44.565388 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.565457 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 18:46:44.566182 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.565979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:40827 (local address 127.0.0.1:60892)
0929 18:46:44.566078 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.566125 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 18:46:44.566289 30588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.157.124:40827: Network error: Client connection negotiation failed: client connection to 127.7.157.124:40827: connect: Connection refused (error 111)
W20260929 18:46:44.571324 30602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:44.571385 30594 server_base.cc:1034] running on GCE node
W20260929 18:46:44.571317 30601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:46:44.571318 30604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:46:44.572189 30594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:46:44.573326 30594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:46:44.574532 30594 hybrid_clock.cc:648] HybridClock initialized: now 1790707604574501 us; error 44 us; skew 500 ppm
Sep 29 18:46:44 dist-test-slave-smpn krb5kdc[30432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707604, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:46:44.577358 30594 init.cc:377] Logged in from keytab as kudu/127.7.157.124@KRBTEST.COM (short username kudu)
I20260929 18:46:44.578298 30594 webserver.cc:460] Webserver started at http://127.7.157.124:44311/ using document root <none> and password file <none>
I20260929 18:46:44.578871 30594 fs_manager.cc:359] Metadata directory not provided
I20260929 18:46:44.578936 30594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:46:44.579172 30594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:46:44.581264 30594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/data/instance:
uuid: "4fa782def06c48f781629442fa46e9bc"
format_stamp: "Formatted at 2026-09-29 18:46:44 on dist-test-slave-smpn"
server_key: "8727ab30f341aa1b441db8314ad99fc1"
server_key_iv: "92ae226f21096521390d084a90b436ca"
server_key_version: "encryptionkey@0"
I20260929 18:46:44.581815 30594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/wal/instance:
uuid: "4fa782def06c48f781629442fa46e9bc"
format_stamp: "Formatted at 2026-09-29 18:46:44 on dist-test-slave-smpn"
server_key: "8727ab30f341aa1b441db8314ad99fc1"
server_key_iv: "92ae226f21096521390d084a90b436ca"
server_key_version: "encryptionkey@0"
I20260929 18:46:44.585538 30594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:46:44.587965 30611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:46:44.589201 30594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:46:44.589323 30594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/data,/tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/wal
uuid: "4fa782def06c48f781629442fa46e9bc"
format_stamp: "Formatted at 2026-09-29 18:46:44 on dist-test-slave-smpn"
server_key: "8727ab30f341aa1b441db8314ad99fc1"
server_key_iv: "92ae226f21096521390d084a90b436ca"
server_key_version: "encryptionkey@0"
I20260929 18:46:44.589419 30594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:46:44.614245 30594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:46:44.617357 30594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:46:44.617543 30594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:46:44.624336 30594 rpc_server.cc:307] RPC server started. Bound to: 127.7.157.124:40827
I20260929 18:46:44.624601 30663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.157.124:40827 every 8 connection(s)
I20260929 18:46:44.625317 30594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/data/info.pb
I20260929 18:46:44.628144 30664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:46:44.631965 30664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:46:44.634581 7797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu as pid 30594
I20260929 18:46:44.634722 7797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLJQtbs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707328618849-7797-0/minicluster-data/master-2/wal/instance
I20260929 18:46:44.635061 7797 external_mini_cluster.cc:1428] Setting key ad0d811ad96b80316e37921b60f3b5eb
Sep 29 18:46:44 dist-test-slave-smpn krb5kdc[30432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707604, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.124@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
Sep 29 18:46:44 dist-test-slave-smpn krb5kdc[30432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.124@KRBTEST.COM
I20260929 18:46:44.645293 30516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.633829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42542 (local address 127.7.157.126:46421)
0929 18:46:44.633953 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.633956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.634959 (+ 1003us) server_negotiation.cc:407] Connection header received
0929 18:46:44.636137 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.636140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.636203 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.636275 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.637301 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.637793 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.638759 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.638981 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.639094 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.641544 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:44.641549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:44.641551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:44.641580 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:44.643480 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.643909 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.643913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.643915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.643999 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.644282 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.644286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.644287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.644472 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:44.644546 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:44.644981 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 18:46:44.645108 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:46:44.646343 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.635757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60902 (local address 127.7.157.124:40827)
0929 18:46:44.636119 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.636130 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.636174 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:46:44.636315 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.636319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.636496 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.636654 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.637402 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.638256 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.639046 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.639435 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.639527 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.641995 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:44.642002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:44.642016 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:44.642041 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:44.643571 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.643948 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.643953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.643954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.644063 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.644305 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.644307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.644308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.644750 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:44.644936 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:44.645126 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 18:46:44.645500 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":228,"server-negotiator.queue_time_us":265,"spinlock_wait_cycles":4352,"thread_start_us":151,"threads_started":1}
I20260929 18:46:44.646341 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.634185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.126:46421 (local address 127.0.0.1:42542)
0929 18:46:44.634769 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.634831 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:44.635853 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:44.636368 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:44.636378 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:44.636717 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:44.637157 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.637169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.637892 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.637896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.638640 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.638649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.639148 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.639152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.639318 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.639870 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:44.639892 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:44.641394 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:44.643579 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.643586 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.643596 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.643820 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.644079 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.644082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.644083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.644202 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.644565 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:44.644572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:44.644750 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 18:46:44.644922 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 18:46:44.645270 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"spinlock_wait_cycles":1280,"thread_start_us":122,"threads_started":1}
I20260929 18:46:44.647693 30664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:46:44 dist-test-slave-smpn krb5kdc[30432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707604, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.124@KRBTEST.COM for kudu/127.7.157.125@KRBTEST.COM
Sep 29 18:46:44 dist-test-slave-smpn krb5kdc[30432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.126@KRBTEST.COM
I20260929 18:46:44.656275 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.648173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.125:35813 (local address 127.0.0.1:36872)
0929 18:46:44.648397 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.648408 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:44.648471 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:44.648856 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:44.648859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:44.649034 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:44.649152 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.649159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.650036 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.650039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.650690 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.650698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.651169 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.651172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.651271 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.651695 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:44.651708 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:44.652947 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:44.654737 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.654740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.654741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.654934 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.655272 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.655275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.655277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.655372 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.655782 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:44.655785 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:44.655874 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 18:46:44.655971 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 18:46:44.656106 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260929 18:46:44.656339 30595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.648280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.7.157.125:35813)
0929 18:46:44.648420 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.648424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.648451 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:46:44.648564 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.648567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.648613 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.648679 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.649258 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.649912 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.650793 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.651041 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.651155 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.653092 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:44.653097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:44.653100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:44.653128 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:44.654595 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.655028 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.655032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.655034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.655175 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.655452 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.655455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.655457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.655675 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:44.655747 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:44.656037 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 18:46:44.656179 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:46:44.656867 30516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.648757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.7.157.126:46421)
0929 18:46:44.648864 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.648868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.648880 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:46:44.648917 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.648920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.648958 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.649053 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.649606 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.650341 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.651122 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.651342 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.651426 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.653523 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:44.653526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:44.653528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:44.653550 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:44.654843 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.655284 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.655287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.655289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.655364 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.655646 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.655649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.655650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.655865 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:44.656122 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:44.656213 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 18:46:44.656755 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:46:44.657250 30664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:46:44.657413 30588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } attempt: 3
I20260929 18:46:44.659706 30513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } attempt: 3
Sep 29 18:46:44 dist-test-slave-smpn krb5kdc[30432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707604, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.125@KRBTEST.COM for kudu/127.7.157.124@KRBTEST.COM
Sep 29 18:46:44 dist-test-slave-smpn krb5kdc[30432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707604, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.124@KRBTEST.COM for kudu/127.7.157.124@KRBTEST.COM
Sep 29 18:46:44 dist-test-slave-smpn krb5kdc[30432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707604, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.157.126@KRBTEST.COM for kudu/127.7.157.124@KRBTEST.COM
Sep 29 18:46:44 dist-test-slave-smpn krb5kdc[30432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.157.125@KRBTEST.COM
I20260929 18:46:44.667850 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.657731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:40827 (local address 127.0.0.1:60920)
0929 18:46:44.657837 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.657851 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:44.657923 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:44.658125 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:44.658128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:44.658291 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:44.658388 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.658395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.659280 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.659283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.659847 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.659856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.660429 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.660433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.660514 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.661368 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:44.661383 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:44.664052 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:44.666280 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.666283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.666284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.666522 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.666793 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.666796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.666798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.666883 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.667334 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:44.667337 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:44.667435 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 18:46:44.667518 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:46:44.667674 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 18:46:44.668462 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.657853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60920 (local address 127.7.157.124:40827)
0929 18:46:44.657951 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.657955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.657979 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:46:44.658005 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.658008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.658041 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.658090 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.658510 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.659194 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.659973 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.660310 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.660714 (+ 404us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.664188 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:44.664192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:44.664194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:44.664219 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:44.666153 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.666622 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.666626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.666628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.666707 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.666965 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.666969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.666971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.667218 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:44.668150 (+ 932us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:44.668212 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 18:46:44.668319 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:46:44.668710 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.657728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:40827 (local address 127.0.0.1:60904)
0929 18:46:44.657823 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.657837 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:44.657908 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:44.658443 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:44.658446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:44.658600 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:44.658682 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.658687 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.660226 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.660229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.660943 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.660955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.661521 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.661524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.661595 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.662641 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:44.662655 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:44.664409 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:44.667418 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.667421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.667423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.667649 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.667890 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.667892 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.667894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.667964 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.668324 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:44.668327 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:44.668399 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 18:46:44.668466 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:46:44.668588 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 18:46:44.669169 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.657937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.7.157.124:40827)
0929 18:46:44.658270 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.658276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.658284 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:46:44.658307 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.658309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.658337 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.658750 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.658799 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.660110 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.661075 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.661426 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.661776 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.664741 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:44.664745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:44.664747 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:44.664773 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:44.667334 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.667743 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.667746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.667748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.667813 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.668053 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.668056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.668057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.668237 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:44.668849 (+ 612us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:44.668917 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 18:46:44.669015 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":127,"threads_started":1}
I20260929 18:46:44.669612 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.660034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.157.124:40827 (local address 127.0.0.1:60936)
0929 18:46:44.660253 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 18:46:44.660266 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:46:44.660336 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:46:44.660990 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:46:44.660993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:46:44.661155 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:46:44.661231 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.661237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.662160 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.662163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.662755 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:46:44.662764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.663227 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.663231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:46:44.663313 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.663695 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:46:44.663706 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:46:44.665250 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:46:44.667433 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.667436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.667438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.667676 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.668233 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:46:44.668236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:46:44.668238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:46:44.668342 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:46:44.669155 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:46:44.669158 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:46:44.669249 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 18:46:44.669326 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:46:44.669455 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260929 18:46:44.669613 30595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.658570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36874 (local address 127.7.157.125:35813)
0929 18:46:44.658661 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.658665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.658676 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:46:44.658704 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.658707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.658742 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.658811 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.659579 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.660291 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.661628 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.662165 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.662238 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.665853 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:44.665857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:44.665859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:44.665884 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:44.667798 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.668233 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.668236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.668238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.668332 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.668624 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.668627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.668629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.668846 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:44.669027 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:44.669387 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 18:46:44.669484 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:46:44.670542 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.660455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.7.157.124:40827)
0929 18:46:44.660789 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.660797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.660808 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:46:44.660835 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.660838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.660873 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.660949 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.661337 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.662050 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.662855 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.663130 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.663217 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.665370 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:44.665374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:44.665376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:44.665402 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:44.667312 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.667794 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.667797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.667799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.667877 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.668450 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.668453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.668454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.669037 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:44.670212 (+ 1175us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:44.670285 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:46:44.670393 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":97,"threads_started":1}
I20260929 18:46:44.672049 30664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc: Bootstrap starting.
I20260929 18:46:44.672055 30588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342: Bootstrap starting.
I20260929 18:46:44.673970 30513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e: Bootstrap starting.
I20260929 18:46:44.674901 30588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:44.674901 30664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:44.675647 7797 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:46:44.675841 30664 log.cc:824] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:44.675841 30588 log.cc:824] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:44.676429 30513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e: Neither blocks nor log segments found. Creating new log.
I20260929 18:46:44.677203 30513 log.cc:824] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:46:44.677047 30487 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "dae107ceee36468aae644236df069a7e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42544
I20260929 18:46:44.679493 30513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e: No bootstrap required, opened a new log
I20260929 18:46:44.679639 30664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc: No bootstrap required, opened a new log
I20260929 18:46:44.679639 30588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342: No bootstrap required, opened a new log
I20260929 18:46:44.682550 30513 raft_consensus.cc:348] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } }
I20260929 18:46:44.682783 30513 raft_consensus.cc:729] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dae107ceee36468aae644236df069a7e, State: Initialized, Role: FOLLOWER
I20260929 18:46:44.683195 30513 consensus_queue.cc:260] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } }
I20260929 18:46:44.683597 30588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } }
I20260929 18:46:44.683812 30588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a99d6e003844e2bae47bd17687fa342, State: Initialized, Role: FOLLOWER
I20260929 18:46:44.683754 30674 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } } }
I20260929 18:46:44.684121 30674 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:46:44.684127 30588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } }
I20260929 18:46:44.684201 30664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } }
I20260929 18:46:44.684428 30664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fa782def06c48f781629442fa46e9bc, State: Initialized, Role: FOLLOWER
I20260929 18:46:44.684636 30513 sys_catalog.cc:564] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:44.684715 30487 raft_consensus.cc:482] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 18:46:44.684717 30664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } }
I20260929 18:46:44.684801 30487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:44.685460 30588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:44.685905 30664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:46:44.686048 30487 raft_consensus.cc:504] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } }
I20260929 18:46:44.687683 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dae107ceee36468aae644236df069a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4fa782def06c48f781629442fa46e9bc"
I20260929 18:46:44.687908 30638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:44.689260 30638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dae107ceee36468aae644236df069a7e in term 1.
I20260929 18:46:44.689678 30562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dae107ceee36468aae644236df069a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5a99d6e003844e2bae47bd17687fa342"
I20260929 18:46:44.689896 30562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:46:44.691201 30562 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dae107ceee36468aae644236df069a7e in term 1.
W20260929 18:46:44.691833 30706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:46:44.691900 30706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:46:44.692070 30675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } } }
I20260929 18:46:44.692117 30461 leader_election.cc:302] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a99d6e003844e2bae47bd17687fa342, dae107ceee36468aae644236df069a7e; no voters:
I20260929 18:46:44.692179 30675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 18:46:44.693351 30466 tablet.cc:2369] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:46:44.695562 30520 master_runner.cc:263] No leader master found from master 5a99d6e003844e2bae47bd17687fa342
I20260929 18:46:44.695631 30520 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:46:44.687016 30487 leader_election.cc:288] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [CANDIDATE]: Term 1 election: Requested vote from peers 5a99d6e003844e2bae47bd17687fa342 (127.7.157.125:35813), 4fa782def06c48f781629442fa46e9bc (127.7.157.124:40827)
I20260929 18:46:44.696147 30445 master_runner.cc:263] No leader master found from master dae107ceee36468aae644236df069a7e
I20260929 18:46:44.696208 30445 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:46:44.692395 30676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } } }
W20260929 18:46:44.696295 30693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:46:44.696364 30693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:46:44.696386 30676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:46:44.696518 30674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:46:44.696758 30674 raft_consensus.cc:686] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [term 1 LEADER]: Becoming Leader. State: Replica: dae107ceee36468aae644236df069a7e, State: Running, Role: LEADER
I20260929 18:46:44.697088 30674 consensus_queue.cc:237] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } }
I20260929 18:46:44.699702 30594 master_runner.cc:263] No leader master found from master 4fa782def06c48f781629442fa46e9bc
I20260929 18:46:44.699765 30594 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:46:44.707782 30713 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dae107ceee36468aae644236df069a7e. Latest consensus state: current_term: 1 leader_uuid: "dae107ceee36468aae644236df069a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } } }
I20260929 18:46:44.707935 30713 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:44.708377 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.698223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.7.157.124:40827)
0929 18:46:44.698330 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.698334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.699569 (+ 1235us) server_negotiation.cc:407] Connection header received
0929 18:46:44.699761 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.699764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.699815 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.699866 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.700672 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.701428 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.702232 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.702597 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.702697 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.704454 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:44.704459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:44.704462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:44.704488 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:44.706271 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.706748 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.706751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.706753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.706848 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.707177 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.707181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.707182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.707411 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:44.707583 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:44.708095 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 18:46:44.708219 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:46:44.708470 30722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260929 18:46:44.708935 30710 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:46:44.709017 30710 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:46:44.709775 30516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.698716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.7.157.126:46421)
0929 18:46:44.698817 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.698822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.700830 (+ 2008us) server_negotiation.cc:407] Connection header received
0929 18:46:44.700985 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.700988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.701045 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.701091 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.702724 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.703437 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.704505 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.704818 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.704894 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.706423 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:44.706427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:44.706429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:44.706456 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:44.708051 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.708492 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.708496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.708498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.708579 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.708868 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.708871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.708873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.709091 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:44.709189 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:44.709525 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 18:46:44.709641 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:46:44.712487 30722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:46:44.718145 30595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:44.699086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.7.157.125:35813)
0929 18:46:44.699188 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.699192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.709252 (+ 10060us) server_negotiation.cc:407] Connection header received
0929 18:46:44.709348 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.709351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.709402 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.709479 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.710767 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.711597 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.712687 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:44.712969 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:44.713042 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:44.714449 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:46:44.714453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:46:44.714456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:46:44.714482 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:46:44.716031 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.716488 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.716492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.716494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.716595 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:46:44.716880 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:46:44.716883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:46:44.716885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:46:44.717104 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:46:44.717234 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:44.717865 (+ 631us) server_negotiation.cc:299] Negotiation successful
0929 18:46:44.717987 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:46:44.718488 30638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc [term 1 FOLLOWER]: Refusing update from remote peer dae107ceee36468aae644236df069a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:46:44.719273 30713 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:46:44.723251 30676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader dae107ceee36468aae644236df069a7e. Latest consensus state: current_term: 1 leader_uuid: "dae107ceee36468aae644236df069a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } } }
I20260929 18:46:44.723394 30676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:46:44.724786 30561 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342 [term 1 FOLLOWER]: Refusing update from remote peer dae107ceee36468aae644236df069a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:46:44.725701 30713 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:46:44.729862 30675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dae107ceee36468aae644236df069a7e. Latest consensus state: current_term: 1 leader_uuid: "dae107ceee36468aae644236df069a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } } }
I20260929 18:46:44.729997 30675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:46:44.732254 30713 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dae107ceee36468aae644236df069a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } } }
I20260929 18:46:44.732381 30713 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:44.732482 30713 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dae107ceee36468aae644236df069a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } } }
I20260929 18:46:44.732563 30713 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e [sys.catalog]: This master's current role is: LEADER
I20260929 18:46:44.732676 30676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dae107ceee36468aae644236df069a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } } }
I20260929 18:46:44.732780 30676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:46:44.732937 30724 mvcc.cc:205] Tried to move back new op lower bound from 7334738348918222848 to 7334738348842741760. Current Snapshot: MvccSnapshot[applied={T|T < 7334738348918222848}]
I20260929 18:46:44.733887 30675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dae107ceee36468aae644236df069a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae107ceee36468aae644236df069a7e" member_type: VOTER last_known_addr { host: "127.7.157.126" port: 46421 } } peers { permanent_uuid: "5a99d6e003844e2bae47bd17687fa342" member_type: VOTER last_known_addr { host: "127.7.157.125" port: 35813 } } peers { permanent_uuid: "4fa782def06c48f781629442fa46e9bc" member_type: VOTER last_known_addr { host: "127.7.157.124" port: 40827 } } }
I20260929 18:46:44.733981 30675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:46:44.734102 30723 mvcc.cc:205] Tried to move back new op lower bound from 7334738348918222848 to 7334738348842741760. Current Snapshot: MvccSnapshot[applied={T|T < 7334738348918222848}]
I20260929 18:46:44.737521 30726 mvcc.cc:205] Tried to move back new op lower bound from 7334738348918222848 to 7334738348842741760. Current Snapshot: MvccSnapshot[applied={T|T < 7334738348918222848}]
I20260929 18:46:44.741683 30722 catalog_manager.cc:1347] Generated new cluster ID: 6f8a320e90c34aab873364b42c7a3ec0
I20260929 18:46:44.741760 30722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:46:44.776835 30722 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:46:44.778403 30722 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:46:44.800241 30722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dae107ceee36468aae644236df069a7e: Generated new TSK 0
I20260929 18:46:44.801172 30722 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:46:44.834024 30669 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:44.832147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60964 (local address 127.7.157.124:40827)
0929 18:46:44.832277 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.832281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.832943 (+ 662us) server_negotiation.cc:407] Connection header received
0929 18:46:44.833139 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.833143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.833196 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.833249 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.833883 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60964: BlockingRecv error: recv got EOF from 127.0.0.1:60964 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 18:46:44.836434 30516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:44.834701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42560 (local address 127.7.157.126:46421)
0929 18:46:44.834814 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.834819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.835511 (+ 692us) server_negotiation.cc:407] Connection header received
0929 18:46:44.835698 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.835701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.835751 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.835800 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.836304 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42560: BlockingRecv error: recv got EOF from 127.0.0.1:42560 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 18:46:44.837910 30595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:44.835248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.7.157.125:35813)
0929 18:46:44.835356 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.835360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.837035 (+ 1675us) server_negotiation.cc:407] Connection header received
0929 18:46:44.837205 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.837208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.837258 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.837306 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.837797 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36882: BlockingRecv error: recv got EOF from 127.0.0.1:36882 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 18:46:44.842247 30595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:44.840779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36896 (local address 127.7.157.125:35813)
0929 18:46:44.840881 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.840885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.841291 (+ 406us) server_negotiation.cc:407] Connection header received
0929 18:46:44.841506 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.841509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.841583 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.841633 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.842143 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36896: BlockingRecv error: recv got EOF from 127.0.0.1:36896 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 18:46:44.847209 30595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:44.845786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36910 (local address 127.7.157.125:35813)
0929 18:46:44.845883 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.845887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.846327 (+ 440us) server_negotiation.cc:407] Connection header received
0929 18:46:44.846499 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.846502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.846551 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.846598 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.847110 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36910: BlockingRecv error: recv got EOF from 127.0.0.1:36910 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 18:46:44.852871 30595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:44.851392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.7.157.125:35813)
0929 18:46:44.851494 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.851498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.851966 (+ 468us) server_negotiation.cc:407] Connection header received
0929 18:46:44.852144 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.852148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.852196 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.852242 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.852758 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36916: BlockingRecv error: recv got EOF from 127.0.0.1:36916 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 18:46:44.861310 30595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:44.859423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36918 (local address 127.7.157.125:35813)
0929 18:46:44.859587 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.859592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.860201 (+ 609us) server_negotiation.cc:407] Connection header received
0929 18:46:44.860451 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.860455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.860517 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.860574 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.861189 (+ 615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36918: BlockingRecv error: recv got EOF from 127.0.0.1:36918 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 18:46:44.877298 30595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:44.875580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36924 (local address 127.7.157.125:35813)
0929 18:46:44.875717 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.875722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.876260 (+ 538us) server_negotiation.cc:407] Connection header received
0929 18:46:44.876450 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.876453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.876504 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.876552 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.877166 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36924: BlockingRecv error: recv got EOF from 127.0.0.1:36924 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 18:46:44.901186 30595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:44.900043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.7.157.125:35813)
0929 18:46:44.900204 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.900209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.900343 (+ 134us) server_negotiation.cc:407] Connection header received
0929 18:46:44.900491 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.900494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.900541 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.900610 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.901058 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36930: BlockingRecv error: recv got EOF from 127.0.0.1:36930 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 18:46:44.936177 30595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:44.935046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36946 (local address 127.7.157.125:35813)
0929 18:46:44.935217 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:44.935221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:44.935332 (+ 111us) server_negotiation.cc:407] Connection header received
0929 18:46:44.935483 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:44.935486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:44.935541 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:44.935612 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:44.936073 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36946: BlockingRecv error: recv got EOF from 127.0.0.1:36946 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 18:46:45.003259 30595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:45.002133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36954 (local address 127.7.157.125:35813)
0929 18:46:45.002273 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:45.002278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:45.002391 (+ 113us) server_negotiation.cc:407] Connection header received
0929 18:46:45.002540 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:45.002543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:45.002591 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:45.002677 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:45.003141 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36954: BlockingRecv error: recv got EOF from 127.0.0.1:36954 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 18:46:45.134341 30595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:45.133194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.7.157.125:35813)
0929 18:46:45.133326 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:45.133332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:45.133489 (+ 157us) server_negotiation.cc:407] Connection header received
0929 18:46:45.133649 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:45.133652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:45.133701 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:45.133769 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:45.134225 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36964: BlockingRecv error: recv got EOF from 127.0.0.1:36964 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 18:46:45.393540 30595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:46:45.392415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36972 (local address 127.7.157.125:35813)
0929 18:46:45.392546 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:45.392550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:45.392765 (+ 215us) server_negotiation.cc:407] Connection header received
0929 18:46:45.392888 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:45.392890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:45.392950 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:45.393024 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:46:45.393425 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36972: BlockingRecv error: recv got EOF from 127.0.0.1:36972 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:46:45.693178 30706 catalog_manager.cc:1259] Loaded cluster ID: 6f8a320e90c34aab873364b42c7a3ec0
I20260929 18:46:45.693264 30706 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342: loading cluster ID for follower catalog manager: success
I20260929 18:46:45.694943 30706 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342: acquiring CA information for follower catalog manager: success
I20260929 18:46:45.696242 30706 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5a99d6e003844e2bae47bd17687fa342: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:46:45.710289 30710 catalog_manager.cc:1259] Loaded cluster ID: 6f8a320e90c34aab873364b42c7a3ec0
I20260929 18:46:45.710376 30710 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc: loading cluster ID for follower catalog manager: success
I20260929 18:46:45.712195 30710 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc: acquiring CA information for follower catalog manager: success
I20260929 18:46:45.713347 30710 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4fa782def06c48f781629442fa46e9bc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:46:45.911258 30795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:45.907622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.7.157.125:35813)
0929 18:46:45.907865 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:45.907868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:45.907883 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:46:45.907996 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:45.907999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:45.908121 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:45.908236 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:46:45.908491 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:45.909001 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:45.909857 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:45.910071 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:45.910150 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:45.910411 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:46:45.910863 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:46:45.910952 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:45.911029 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 18:46:45.911108 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":72,"threads_started":1}
I20260929 18:46:45.917922 30801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:45.914262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.7.157.124:40827)
0929 18:46:45.914480 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:45.914484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:45.914499 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:46:45.914633 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:45.914635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:45.914754 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:45.914851 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:46:45.915009 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:45.915545 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:45.916351 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:45.916616 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:45.916697 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:45.917047 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:46:45.917542 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:46:45.917647 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:45.917695 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 18:46:45.917748 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":69,"threads_started":1}
I20260929 18:46:45.924957 30795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:45.921512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.7.157.125:35813)
0929 18:46:45.921634 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:45.921638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:45.921651 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:46:45.921699 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:45.921702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:45.921750 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:45.921810 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:46:45.922145 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:45.922678 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:45.923779 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:45.924069 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:45.924164 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:45.924443 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:46:45.924534 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:46:45.924599 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:45.924747 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 18:46:45.924807 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:46:45.925028 30801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:45.921194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.7.157.124:40827)
0929 18:46:45.921305 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:45.921309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:45.921524 (+ 215us) server_negotiation.cc:407] Connection header received
0929 18:46:45.921702 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:45.921705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:45.921749 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:45.921793 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:46:45.922219 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:45.922985 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:45.923912 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:45.924215 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:45.924304 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:45.924577 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:46:45.924672 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:46:45.924718 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:45.924864 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 18:46:45.924919 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:46:45.925832 30808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:46:45.920956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56978 (local address 127.7.157.126:46421)
0929 18:46:45.921196 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 18:46:45.921200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:46:45.921216 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:46:45.921416 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:46:45.921420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:46:45.921619 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:46:45.921743 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:46:45.922169 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:45.922992 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:45.924087 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:46:45.924411 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:46:45.924512 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:46:45.924789 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:46:45.925180 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:46:45.925264 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:46:45.925624 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 18:46:45.925689 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
I20260929 18:46:45.928903 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 30445
I20260929 18:46:45.934446 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 30520
I20260929 18:46:45.939128 7797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLJQtbs/build/debug/bin/kudu with pid 30594
2026-09-29T18:46:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2995 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5980 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (277312 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST